| panscient | - /abcd/
- /abcd/great12345.php
- great12345.php
- /great12345.php
- http://www.ibankdesign.com/great12345.php
- http://ibankdesign.com/great12345.php
- http://ibankdesign.com/abcd/great12345.php
- http://www.ibankdesign.com/abcd/great12345.php
- /home/ibankdes/public_html/great12345.php
- /home/ibankdes/public_html/abcd/
|
| Baiduspider+(+http://www.baidu.com/search/spider.htm) | - All subpages are allowed to crawl
|
| Baiduspider+ | - All subpages are allowed to crawl
|
| BaiDuSpider | - All subpages are allowed to crawl
|
| asterias | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| BackDoorBot/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Black Hole | - All subpages are allowed to crawl
|
| BlowFish/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| BotALot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| BuiltBotTough | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Bullseye/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| BunnySlippers | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Cegbfeieh | - All subpages are allowed to crawl
|
| CheeseBot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| CherryPicker | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| CherryPickerElite/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| CherryPickerSE/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| CopyRightCheck | - All subpages are allowed to crawl
|
| cosmos | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Crescent | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Crescent Internet ToolPak HTTP OLE Control v.1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| DittoSpyder | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| EmailCollector | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| EmailSiphon | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| EmailWolf | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| EroCrawler | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| ExtractorPro | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Foobot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Harvest/1.5 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| hloader | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| httplib | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| humanlinks | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| InfoNaviRobot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| JennyBot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Kenjin Spider | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Keyword Density/0.9 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| LexiBot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| libWeb/clsHTTP | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| LinkextractorPro | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| LinkScan/8.1a Unix | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| LinkWalker | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| LNSpiderguy | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| lwp-trivial | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| lwp-trivial/1.34 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Mata Hari | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Microsoft URL Control - 5.01.4511 | - All subpages are allowed to crawl
|
| Microsoft URL Control - 6.00.8169 | - All subpages are allowed to crawl
|
| MIIxpc | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| MIIxpc/4.2 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Mister PiX | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| moget | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| moget/2.1 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| mozilla/4 | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; BullsEye; Windows 95) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) | - All subpages are allowed to crawl
|
| Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) | - All subpages are allowed to crawl
|
| mozilla/5 | - All subpages are allowed to crawl
|
| NetAnts | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| NICErsPRO | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Offline Explorer | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Openfind | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Openfind data gathere | - All subpages are allowed to crawl
|
| ProPowerBot/2.14 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| ProWebWalker | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| QueryN Metasearch | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| RepoMonkey | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| RepoMonkey Bait & Tackle/v1.01 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| RMA | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| SiteSnagger | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| SpankBot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| spanner | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| suzuran | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Szukacz/1.4 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Teleport | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| TeleportPro | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Telesoft | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| The Intraformant | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| TheNomad | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| TightTwatBot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Titan | - All subpages are allowed to crawl
|
| toCrawl/UrlDispatcher | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| True_Robot | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| True_Robot/1.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| turingos | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| URLy Warning | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| VCI | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| VCI WebViewer VCI WebViewer Win32 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Web Image Collector | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebAuto | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebBandit | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebBandit/3.50 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebCopier | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebEnhancer | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebmasterWorldForumBot | - All subpages are allowed to crawl
|
| WebSauger | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Website Quester | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Webster Pro | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebStripper | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebZip | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WebZip/4.0 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Wget | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Wget/1.5.3 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Wget/1.6 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| WWW-Collector-E | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Xenu's | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Xenu's Link Sleuth 1.1c | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Zeus | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| Zeus 32297 Webster Pro V2.9 Win32 | - All subpages are allowed to crawl
- All subpages are denied to crawl
|
| aipbot | - All subpages are denied to crawl
|
| ia_archiver | - All subpages are denied to crawl
|
| Alexibot | - All subpages are denied to crawl
|
| Aqua_Products | - All subpages are denied to crawl
|
| b2w/0.1 | - All subpages are denied to crawl
|
| becomebot | - All subpages are denied to crawl
|
| Bookmark search tool | - All subpages are denied to crawl
|
| BotRightHere | - All subpages are denied to crawl
|
| Copernic | - All subpages are denied to crawl
|
| FairAd Client | - All subpages are denied to crawl
|
| Fasterfox | - All subpages are denied to crawl
|
| Flaming AttackBot | - All subpages are denied to crawl
|
| Gaisbot | - All subpages are denied to crawl
|
| GetRight/4.2 | - All subpages are denied to crawl
|
| HTTrack 3.0 | - All subpages are denied to crawl
|
| IconSurf | - All subpages are denied to crawl
- /favicon.ico
|
| Iron33/1.0.2 | - All subpages are denied to crawl
|
| larbin | - All subpages are denied to crawl
|
| MSIECrawler | - All subpages are denied to crawl
|
| Openbot | - All subpages are denied to crawl
|
| Openfind data gatherer | - All subpages are denied to crawl
|
| Oracle Ultra Search | - All subpages are denied to crawl
|
| PerMan | - All subpages are denied to crawl
|
| psbot | - All subpages are denied to crawl
|
| Python-urllib | - All subpages are denied to crawl
|
| Radiation Retriever 1.1 | - All subpages are denied to crawl
|
| searchpreview | - All subpages are denied to crawl
|
| SurveyBot | - All subpages are denied to crawl
|
| TurnitinBot | - All subpages are denied to crawl
|
| TurnitinBot/1.5 | - All subpages are denied to crawl
|
| URL Control | - All subpages are denied to crawl
|
| URL_Spider_Pro | - All subpages are denied to crawl
|
| WebCapture 2.0 | - All subpages are denied to crawl
|
| WebCopier v.2.2 | - All subpages are denied to crawl
|
| WebCopier v3.2a | - All subpages are denied to crawl
|
| WebZIP/4.21 | - All subpages are denied to crawl
|
| WebZIP/5.0 | - All subpages are denied to crawl
|
| wget | - All subpages are denied to crawl
|
| Zeus Link Scout | - All subpages are denied to crawl
- /cgi-bin/
|