The Best Robots.txt file to use

Share tips and tricks with the AOM user community.
Post Reply
d2s
Posts: 7
Joined: Fri Apr 11, 2014 10:47 pm

The Best Robots.txt file to use

Post by d2s » Wed Apr 23, 2014 11:18 pm

Can somebody post the best Robots.txt file to use that blocks the most common bots that affect your website. I know there are thousands out there but does anyone know the main culprits
Associate-O-Matic is changing. For the better. 8)

Please CLICK HERE for our important announcement.

goodhuman
Experience Level 1
Experience Level 1
Posts: 10
Joined: Sun Aug 17, 2014 12:09 am

Re: The Best Robots.txt file to use

Post by goodhuman » Sun Aug 17, 2014 12:25 am

d2s wrote:Can somebody post the best Robots.txt file to use that blocks the most common bots that affect your website. I know there are thousands out there but does anyone know the main culprits
why do you want to block bots this will stop search engines from indexing your website and first you need to identify what bots you want to block playing with robots.txt is bit risky thing.
Associate-O-Matic is changing. For the better. 8)

Please CLICK HERE for our important announcement.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests