CRAN/E | robotstxt

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Installation

About

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

docs.ropensci.org/robotstxt/
github.com/ropensci/robotstxt
Bug report File report

Key Metrics

Version 0.7.13
R ≥ 3.0.0
Published 2020-09-03 1332 days ago
Needs compilation? no
License MIT
License File
CRAN checks robotstxt results

Downloads

Yesterday 128 +13%
Last 7 days 831 +66%
Last 30 days 2.292 -6%
Last 90 days 7.004 -11%
Last 365 days 33.177 +1%

Maintainer

Maintainer

Peter Meissner

retep.meissner@gmail.com

Authors

Peter Meissner

aut / cre

Kun Ren

aut / cph

(Author and copyright holder of list_merge.R.)

Oliver Keys

ctb

(original release code review)

Rich Fitz John

ctb

(original release code review)

Material

README
NEWS
Reference manual
Package source

In Views

WebTechnologies

Vignettes

using_robotstxt

macOS

r-release

arm64

r-oldrel

arm64

r-release

x86_64

r-oldrel

x86_64

Windows

r-devel

x86_64

r-release

x86_64

r-oldrel

x86_64

Old Sources

robotstxt archive

Depends

R ≥ 3.0.0

Imports

stringr ≥ 1.0.0
httr ≥ 1.0.0
spiderbar ≥ 0.2.0
future ≥ 1.6.2
future.apply ≥ 1.0.0
magrittr
utils

Suggests

knitr
rmarkdown
dplyr
testthat
covr

Reverse Imports

polite
ralger

Reverse Suggests

newsanchor
spiderbar
vosonSML
webchem