first time on Serverfault.

I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a variety of websites. It's been working for weeks on my server but sometime in the last couple of days Myspace has seemed to begun redirecting me to Google.com (?) and when I ping them I get an address that returns a 403 in my browser:

64 bytes from icmp_seq=1 ttl=242 time=31.6 ms
64 bytes from icmp_seq=2 ttl=242 time=31.5 ms
64 bytes from icmp_seq=3 ttl=242 time=31.6 ms

To be honest I've been having my bot ignore robots.txt and pretend to be a Mac user by dressing up in my own user-agent. I figured it wasn't unethical since I'm not spamming at all.

Just for kicks, I ran mechanize with Googlebot 2.1's useragent to try connecting to Myspace.com really quickly and got Google.com back as well. Different address now when I'm pinging it after that, but also a 403 error:

64 bytes from icmp_seq=1 ttl=243 time=37.1 ms
64 bytes from icmp_seq=2 ttl=243 time=37.5 ms
64 bytes from icmp_seq=3 ttl=243 time=37.3 ms

So logic tells me they blocked me. I'm wondering if there is a workaround for this problem. And if they have, why are they redirecting me to Google?

I've also read others have had this happen as a result of some temporary geo-specific DNS problems. Seems likely, since both companies have so many servers. My bot has no problems running locally from my computer. Possible?

Edit: If it's relevant, he doesn't run too freqeuently, especially not in the past few days. I haven't had him on a schedule of any kind since I'm still developing him. There have been two occasions where I've run him for an hour or so nonstop, none of which have been in the past week, and didn't have this problem. It seems to have just arose.

closed as too localized by pauska, Scott Pack, Tom O'Connor, Chris S, Chopper3 Jan 5 '12 at 14:10

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 9
    You just got served. Ignoring robots.txt =! unethical. Contact MySpace directly if you want to data mine them. – pauska Jan 5 '12 at 12:47
  • It isn't unethical? Will try contacting them. – Artur Sapek Jan 5 '12 at 12:53
  • 4
    Yes it is, that's what it's there for. – Chopper3 Jan 5 '12 at 13:01
  • 2
    @ArturSapek gah, I meant =, not !=. – pauska Jan 5 '12 at 13:58

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic.

An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just blackhole your traffic (that's what my monitoring does). But simply redirecting the traffic to another site would also be a reasonable response and returning a 403 error code is apparently their polite way of telling you to go away.

Not the answer you're looking for? Browse other questions tagged or ask your own question.