PHP class for parse all directives from robots.txt files according to specifications
-
Updated
Jan 28, 2024 - DIGITAL Command Language
#
robots-txt-parser
Here are 6 public repositories matching this topic...
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
Jun 23, 2023 - JavaScript
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
web-scraping robots-txt web-crawling hacktoberfest robots-exclusion-standard robots-exclusion-protocol robots-txt-parser
-
Updated
Nov 6, 2024 - PHP
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
-
Updated
Dec 2, 2024 - Dart
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
-
Updated
Jul 22, 2024 - Python
Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.
-
Updated
Jan 6, 2025 - TypeScript
Improve this page
Add a description, image, and links to the robots-txt-parser topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt-parser topic, visit your repo's landing page and select "manage topics."