support for reading remote robots.txt files

Support for reading remote robots.txt files as discussed in #2

updated urls and licenses

Updated URLs about https://serpwings.com and Licensing information.

Version with Crawl_delay Support Latest

List of Updates

• Updated Documentation
• Support for Crawl_delay: See #1 for more details
• Working Example. See #3 for more details

initial release for pipy

Similar to v0.0.1 but now available to pipy for public release.

Initial Release

Basic Robots.txt File creation Functionality

You can do the followings.

• Add Header/footer Comments
• Add Creation Date
• Add and Remove Allowed Items/Pages/Routes
• Add and Remove Disallowed Items/Pages/Routes
• Add/Remove Sitemaps
• Include Fancy ASCII Images