-4

first time on Serverfault.

I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a variety of websites. It's been working for weeks on my server but sometime in the last couple of days Myspace has seemed to begun redirecting me to Google.com (?) and when I ping them I get an address that returns a 403 in my browser: 

64 bytes from 63.135.80.224: icmp_seq=1 ttl=242 time=31.6 ms
64 bytes from 63.135.80.224: icmp_seq=2 ttl=242 time=31.5 ms
64 bytes from 63.135.80.224: icmp_seq=3 ttl=242 time=31.6 ms

To be honest I've been having my bot ignore robots.txt and pretend to be a Mac user by dressing up in my own user-agent. I figured it wasn't unethical since I'm not spamming at all.

Just for kicks, I ran mechanize with Googlebot 2.1's useragent to try connecting to Myspace.com really quickly and got Google.com back as well. Different address now when I'm pinging it after that, but also a 403 error:

64 bytes from 216.178.46.224: icmp_seq=1 ttl=243 time=37.1 ms
64 bytes from 216.178.46.224: icmp_seq=2 ttl=243 time=37.5 ms
64 bytes from 216.178.46.224: icmp_seq=3 ttl=243 time=37.3 ms

So logic tells me they blocked me. I'm wondering if there is a workaround for this problem. And if they have, why are they redirecting me to Google?

I've also read others have had this happen as a result of some temporary geo-specific DNS problems. Seems likely, since both companies have so many servers. My bot has no problems running locally from my computer. Possible?

Edit: If it's relevant, he doesn't run too freqeuently, especially not in the past few days. I haven't had him on a schedule of any kind since I'm still developing him. There have been two occasions where I've run him for an hour or so nonstop, none of which have been in the past week, and didn't have this problem. It seems to have just arose.

Improve this question
4
  • 9
    You just got served. Ignoring robots.txt =! unethical. Contact MySpace directly if you want to data mine them.
    – pauska
    Jan 5, 2012 at 12:47
  • It isn't unethical? Will try contacting them.
    – Artur Sapek
    Jan 5, 2012 at 12:53
  • 4
    Yes it is, that's what it's there for.
    – Chopper3
    Jan 5, 2012 at 13:01
  • 2
    @ArturSapek gah, I meant =, not !=.
    – pauska
    Jan 5, 2012 at 13:58

1 Answer 1

Reset to default
4

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic.

An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just blackhole your traffic (that's what my monitoring does). But simply redirecting the traffic to another site would also be a reasonable response and returning a 403 error code is apparently their polite way of telling you to go away.

Improve this answer
0

Not the answer you're looking for? Browse other questions tagged .