0

We are experiencing a problem with scraping on our e-commerce platform.

It's not the scraping that is the problem per se, it's that whoever is doing it is spoofing a stale referrer full of our custom analytics parameters. This is playing havoc with our in-house analytics package.

All of the IPs are originating from 1e100.net, i.e., Google IP addresses.

I want to block all these addresses from I'm concerned that it's possible I may inadvertently stop Google company resources from accessing our site. These might include, for example, resources that index our site, or other resources owned by the Adwords/Analytics team.

Is there any way to differentiate between Google App Engine traffic (which is what I suspect this traffic is), and Google company traffic.

1 Answer 1

2

Google maintains a list of netblocks in a TXT record _netblocks.google.com these are the netblocks that GAE may use to make requests. The current answer looks like this:

gbeech@ny-man01:~$ dig -t TXT _netblocks.google.com @ns1.google.com

; <<>> DiG 9.7.0-P1 <<>> -t TXT _netblocks.google.com @ns1.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64157
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;_netblocks.google.com.         IN      TXT

;; ANSWER SECTION:
_netblocks.google.com.  3600    IN      TXT     "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all"

;; Query time: 33 msec
;; SERVER: 216.239.32.10#53(216.239.32.10)
;; WHEN: Wed May  9 22:41:45 2012
;; MSG SIZE  rcvd: 254
1
  • Great, thanks! But I still don't understand the relationship between GAE and Google company resources. For example, will Google every do any indexing using GAE IPs? Or will it ever check destination URLs in Adwords using GAE IPs? Or will it ever use GAE IPs for scraping, to help with the assignment of quality score in Adwords? etc etc.
    – Edwardr
    May 10, 2012 at 8:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .