first time on Serverfault.
I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a variety of websites. It's been working for weeks on my server but sometime in the last couple of days Myspace has seemed to begun redirecting me to Google.com (?) and when I ping them I get an address that returns a 403 in my browser:
64 bytes from 220.127.116.11: icmp_seq=1 ttl=242 time=31.6 ms 64 bytes from 18.104.22.168: icmp_seq=2 ttl=242 time=31.5 ms 64 bytes from 22.214.171.124: icmp_seq=3 ttl=242 time=31.6 ms
To be honest I've been having my bot ignore robots.txt and pretend to be a Mac user by dressing up in my own user-agent. I figured it wasn't unethical since I'm not spamming at all.
Just for kicks, I ran mechanize with Googlebot 2.1's useragent to try connecting to Myspace.com really quickly and got Google.com back as well. Different address now when I'm pinging it after that, but also a 403 error:
64 bytes from 126.96.36.199: icmp_seq=1 ttl=243 time=37.1 ms 64 bytes from 188.8.131.52: icmp_seq=2 ttl=243 time=37.5 ms 64 bytes from 184.108.40.206: icmp_seq=3 ttl=243 time=37.3 ms
So logic tells me they blocked me. I'm wondering if there is a workaround for this problem. And if they have, why are they redirecting me to Google?
I've also read others have had this happen as a result of some temporary geo-specific DNS problems. Seems likely, since both companies have so many servers. My bot has no problems running locally from my computer. Possible?
Edit: If it's relevant, he doesn't run too freqeuently, especially not in the past few days. I haven't had him on a schedule of any kind since I'm still developing him. There have been two occasions where I've run him for an hour or so nonstop, none of which have been in the past week, and didn't have this problem. It seems to have just arose.