PDA

View Full Version : Baidu Spiders?


corsair23
02-21-2012, 12:13 PM
Not sure why I noticed it, but I noticed that there were a LOT of "guests" with the IP address of 180.76.5.XXX

So, out of curiosity I did a Google search on 180.76.5 and one of the hits was:

https://www.vbulletin.com/forum/showthread.php/387062-Baidu-spiders


Not sure if they really affect our web site's performance but just now for instance, out of 75 "people" online, 36 of them were Baidu spiders :confused: - So almost 1/2. I know there are Google and Yahoo and other bots out there but typically you only see a couple instances of each at any one time.

Anyway, I'm not web guru but thought I'd bring it up...

Hulk
02-21-2012, 12:23 PM
Looks like we just need to create a robot.txt file. I'll work on this.

corsair23
02-21-2012, 12:32 PM
:cool:

Looks like 180.76.6.xxx might be another one?

http://www.forumpostersunion.com/showthread.php?p=86441


The two together added up to 43 out of the 75 users online just a minute ago when I counted. Seems like they are the bain of many web folk :(

60wag
02-21-2012, 02:20 PM
I know nothing about bots but have noticed the high number of users lately. How does the .txt file keep the bots out?

corsair23
02-21-2012, 02:56 PM
I know nothing about bots but have noticed the high number of users lately. How does the .txt file keep the bots out?

Matt can probably explain it but this was a link within the first link I posted:

http://www.robotstxt.org/robotstxt.html

simps80
02-22-2012, 10:02 PM
dang 8:56pm
Currently Active Users: 104 (12 members and 92 guests)
google spider's goin' nuts!!

corsair23
02-23-2012, 01:27 AM
dang 8:56pm
Currently Active Users: 104 (12 members and 92 guests)
google spider's goin' nuts!!

no doubt...

12:26am
103 users (3 members and 100 guests)

I counted 22 Google spiders and at least 50 Baidu spiders...Oh, and 1 MSN bot :hill:

corsair23
04-26-2012, 03:10 PM
Seems like the spiders are getting worse...

Just checked and out of the 61 users online a few minutes ago, 52 were spiders with IP addresses starting with 180.76.5 & 180.76.6 :(

wesintl
04-26-2012, 04:42 PM
so... they may suck but they are really only sucking bandwith which isn't an issue for us right now. :MEH: Not sure why it's got you so worried :confused:

Some don't respect the robot.txt file so you have to block them at htacces or other creative ways

corsair23
04-26-2012, 06:06 PM
Spiders scares me :hill:

Hulk
04-30-2012, 10:29 PM
Huh. I wonder if I can block them at the server level.

Hulk
11-30-2012, 01:18 AM
Hmmm. Looks like I turned up the volume on the robots.txt file a little too high. I just fixed it. :o

Not sure if it ever did any good anyway.