Here at the 10 Million Dollar Page, we make use of a Bot process to gather information from our clients. Unlike traditional Bots, the 10milliondollarpage bot will only query our client's site, as specified in our Purchasing Terms and Conditions.
Since this is a mutual agreement, our Bot does not conform to the rules as other Bots. We do not look for a robots.txt file. We only query 2 files per client, the XML file and the Grid image.
If you have any questions or concerns with our Bot, please contact us