Google Cloud App Engine hosted page title extractor returns 403 Forbidden in cases

Question

I have a URL shortener/traffic analysis service hosted in a Google App Engine (GAE) project on Google Cloud (GC). When a URL is submitted for shortening, it performs a Python urllib2.urlopen to extract the page and get its title, which is uses to generate a social media post containing the shortened link. Some sites are returning a 403 Forbidden on the urllib2.urlopen command, and are unable to generate the body of the social media post from the page title as a result. Note that this only occurs when trying to load the page to get the page title; the shortened URL generated works correctly in my browser and 302s to the target page correctly.

My guess is that page requests coming from the GC IP ranges are blacklisted by the sites I'm trying to link to as a source of possible bot or other non-human or malicious traffic.

Does GC have a whitelisting service to which I could apply, or is this caused by something else?

Answer 1

As per Outbound Requests, App Engine uses the URL Fetch service to issue outbound requests. Requests from your application are in effect being proxied through Google's URL Fetch service. Other sites can often rather easily identify the source of these requests. In addition, App Engine will append an identifier string to the User-Agent header to allow servers to identify App Engine requests as per Request headers.

There is no whitelisting service available that will allow your App Engine application to fetch URLs and get 2xx responses. Requesting a given URL and parsing the HTML response for some portion of its content is commonly referred to as scraping and most large web sites have stern policies concerning this type of interaction.

Facebook, for example

Facebook's Terms of Service explicitly disallow scraping entirely stating the following:

You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our prior permission.

Facebook does have its Graph API which is the main way of getting data in and out of Facebook's social graph. This requires registering an app with them and getting a app secret. This way you can provide an appsecret_proof to Graph API calls for authentication. Then, you can use the URL call to essentially get a Facebook object (shares, comments, etc.) by providing an external Facebook URL.

The general case

Most other social media sites have their own APIs as well. This is because they want to decide how external applications such as yours access their content and content given to them by their users. You must be sure to respect each site's terms of service if you are to access their resources.