CRAN/E | spiderbar

spiderbar

Parse and Test Robots Exclusion Protocol Files and Rules

Installation

About

The 'Robots Exclusion Protocol' documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' C++ library for processing these 'robots.txt' files.

github.com/hrbrmstr/spiderbar
Bug report File report

Key Metrics

Version 0.2.5
R ≥ 3.2.0
Published 2023-02-11 447 days ago
Needs compilation? yes
License MIT
License File
CRAN checks spiderbar results

Downloads

Yesterday 93 0%
Last 7 days 511 -40%
Last 30 days 2.254 +0%
Last 90 days 6.578 -14%
Last 365 days 32.426 +1%

Maintainer

Maintainer

Bob Rudis

bob@rud.is

Authors

Bob Rudis

aut / cre

(bob@rud.is)

SEOmoz
Inc

aut

Material

Reference manual
Package source

In Views

WebTechnologies

macOS

r-release

arm64

r-oldrel

arm64

r-release

x86_64

r-oldrel

x86_64

Windows

r-devel

x86_64

r-release

x86_64

r-oldrel

x86_64

Old Sources

spiderbar archive

Depends

R ≥ 3.2.0

Imports

Rcpp

Suggests

covr
robotstxt
tinytest

LinkingTo

Rcpp

Reverse Imports

robotstxt