Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are experiencing a problem with scraping on our e-commerce platform.

It's not the scraping that is the problem per se, it's that whoever is doing it is spoofing a stale referrer full of our custom analytics parameters. This is playing havoc with our in-house analytics package.

All of the IPs are originating from 1e100.net, i.e., Google IP addresses.

I want to block all these addresses from I'm concerned that it's possible I may inadvertently stop Google company resources from accessing our site. These might include, for example, resources that index our site, or other resources owned by the Adwords/Analytics team.

Is there any way to differentiate between Google App Engine traffic (which is what I suspect this traffic is), and Google company traffic.

share|improve this question

1 Answer 1

Google maintains a list of netblocks in a TXT record _netblocks.google.com these are the netblocks that GAE may use to make requests. The current answer looks like this:

gbeech@ny-man01:~$ dig -t TXT _netblocks.google.com @ns1.google.com

; <<>> DiG 9.7.0-P1 <<>> -t TXT _netblocks.google.com @ns1.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64157
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;_netblocks.google.com.         IN      TXT

;; ANSWER SECTION:
_netblocks.google.com.  3600    IN      TXT     "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all"

;; Query time: 33 msec
;; SERVER: 216.239.32.10#53(216.239.32.10)
;; WHEN: Wed May  9 22:41:45 2012
;; MSG SIZE  rcvd: 254
share|improve this answer
    
Great, thanks! But I still don't understand the relationship between GAE and Google company resources. For example, will Google every do any indexing using GAE IPs? Or will it ever check destination URLs in Adwords using GAE IPs? Or will it ever use GAE IPs for scraping, to help with the assignment of quality score in Adwords? etc etc. –  Edwardr May 10 '12 at 8:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.