Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We are experiencing a problem with scraping on our e-commerce platform.

It's not the scraping that is the problem per se, it's that whoever is doing it is spoofing a stale referrer full of our custom analytics parameters. This is playing havoc with our in-house analytics package.

All of the IPs are originating from, i.e., Google IP addresses.

I want to block all these addresses from I'm concerned that it's possible I may inadvertently stop Google company resources from accessing our site. These might include, for example, resources that index our site, or other resources owned by the Adwords/Analytics team.

Is there any way to differentiate between Google App Engine traffic (which is what I suspect this traffic is), and Google company traffic.

share|improve this question

Google maintains a list of netblocks in a TXT record these are the netblocks that GAE may use to make requests. The current answer looks like this:

gbeech@ny-man01:~$ dig -t TXT

; <<>> DiG 9.7.0-P1 <<>> -t TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64157
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;         IN      TXT

;; ANSWER SECTION:  3600    IN      TXT     "v=spf1 ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ?all"

;; Query time: 33 msec
;; WHEN: Wed May  9 22:41:45 2012
;; MSG SIZE  rcvd: 254
share|improve this answer
Great, thanks! But I still don't understand the relationship between GAE and Google company resources. For example, will Google every do any indexing using GAE IPs? Or will it ever check destination URLs in Adwords using GAE IPs? Or will it ever use GAE IPs for scraping, to help with the assignment of quality score in Adwords? etc etc. – Edwardr May 10 '12 at 8:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.