58

In Poland, it is common for mobile ISPs to offer plans with limited amount of bandwidth per month, with exclusion of some popular apps. So for example all traffic from YouTube is not counted towards the data cap.

Aside from net neutrality issues, I am wondering how is this achieved in the HTTPS age? How does the ISP know which packets to count towards the data cap?

I know it could be done with just looking at the IP address, but YouTube has a ton of IPs, and I suspect they change all the time. Plus, I wouldn't be surprised if some of YouTube's IPs are shared with other Google services, which are not uncapped by the ISPs...

Improve this question
19
  • 6
    Youtuve has whatever IP address the DNS (of your provider) returns. Easy to change that to local IP port proxies and not account for them.
    – TomTom
    Jan 5, 2020 at 11:16
  • 7
    @TomTom Given things like certificate pinning and DNS over HTTPS, they may be whitelisting IP ranges instead.
    – ceejayoz
    Jan 5, 2020 at 18:44
  • 4
    What's the advantage to the ISP of doing this? Is it actually cheaper for them to transfer data from Youtube than Serverfault?
    – user3490
    Jan 6, 2020 at 16:14
  • 9
    @user3490 Large companies frequently pay to have their sites exempted, which offsets the bandwidth costs. Sometimes, popular sites are exempted because otherwise a significant portion of users would exceed the caps. Exempting them allows the caps to stay low enough to scare most users into voluntarily limiting usage while avoiding having too many people exceed them. Too many people exceeding the caps leads to lots of complaints, which leads to pressure to eliminate the caps (and all the money ISPs earn from them).
    – bta
    Jan 6, 2020 at 23:17
  • 9
    @user3490 This is the kind of crap you get when net neutrality goes out the window.
    – Mast
    Jan 7, 2020 at 6:53

6 Answers 6

Reset to default
46

Firstly they know the YouTube IP address.

ISP's have an IP database. For example YouTube's ASN is AS15169. On the server side they would make a grouping for each service. One of them is the default grouping and this is the billing group. When you make use of default group, that usage is recorded in the system.

For example a few YouTube addresses are listed below.

root@server ~>whois -h whois.radb.net -- '-i origin AS15169' | grep ^route
route:      192.179.147.0/24
route:      192.179.148.0/23
route:      192.179.148.0/24
route:      192.179.149.0/24
route:      192.179.150.0/23
route:      192.179.150.0/
...
route6:     2607:f8b0:4016::/48
route6:     2604:31C0::/32
route6:     2620:33:c000::/48
route6:     2607:f8b0:4000::/48
route6:     2404:f340::/32

When you are trying to reach YouTube or other YouTube services (Google video storage) your phone will try to reach these IP addresses.

The ISP checks the IP address, and if it is inside the YouTube group, they don't apply charge at this group.

Another option is checking the SNI header at the initial HTTP connection. When you make a connection with HTTPS sites not all the data is encrypted.

For example, when you make a search on Google, you can see the URL in your browser like this: https://www.google.com/search?q=hello+world.

Encrypted data is /search?q=hello+world and all page content. Now you are reaching a site like www.google.com, but they don't know which page or the content inside of that page.

Some ISPs use SNI for this. For example in Turkey this method is used for making specific internet packages like 5GB internet+4GB Spotify or 7GB internet with unlimited WhatsApp. Also they use SNI for banning websites. Some websites use the same IP addresses like wikimedia.com or wikipedia.org. If they try to block Wikipedia with an IP addresses they block all Wikimedia services.

Improve this answer
9
  • 1
    Why would they query DNS if they already have the SNI directly from the traffic?
    – Esa Jokinen
    Jan 5, 2020 at 13:07
  • 3
    ok. I make a edit for better explanation . Hopely it is better.
    – Ahmet Özer
    Jan 5, 2020 at 17:25
  • 1
    "When you make a connection with https sites not all the data is encrypted"... except if it's using preloaded HSTS, like YouTube, in which case all HTTP traffic is encrypted.
    – ArtOfCode
    Jan 6, 2020 at 19:04
  • 3
    HSTS will not protect the HTTPS handshake, nor the DH exchange, so you are incorrect in your assertion that "all" HTTP traffic is encrypted when HSTS is active, @ArtOfCode
    – Kamilion
    Jan 7, 2020 at 0:18
  • 1
    That's not HTTP traffic, @Kamilion. All traffic using the HTTP protocol is encrypted. Not all traffic using the TLS protocol is encrypted.
    – ArtOfCode
    Jan 7, 2020 at 0:58
64

HTTPS obscures the content of the traffic, but not the endpoints. So, for instance, my ISP does not know that I'm responding to this particular question, because I'm using HTTPS, but they do still know that I'm accessing content on serverfault.com port 443.

In specific cases such as the ISP/Neflix partnerships you describe, it's also common for Netflix to co-locate one of their endpoints in the ISP's data center, which then operates similarly to a CDN - when you connect to Netflix, you get the Netflix server on the ISP's own network, which makes it even easier for them to track it for purposes of the deal, since your traffic never leaves their own network. (The co-located server still needs to get the video streams from another Netflix server outside of the ISP's network, of course, but funneling everything through the co-located server allows them to cache data, use dedicated connections, aggregate streams, etc. to reduce their costs and/or pass some of those costs back to Netflix as part of the partnership deal.)

Improve this answer
6
  • 9
    +1. Let me emphasize: they don’t just know the target IP, but they also know the destination hostname (serverfault.com) because it’s sent unencrypted — and the server can use it to choose how to decrypt the traffic.
    – Blaisorblade
    Jan 5, 2020 at 17:48
  • 1
    @Blaisorblade There are means to hide hostname (like DNSSEC). Hiding endpoint is much harder and need some kind of proxy.
    – val - disappointed in SE
    Jan 5, 2020 at 18:07
  • 12
    @valsaysReinstateMonica I am talking about SNI, a feature of HTTPS. It sounds confusing that DNSSEC would affect HTTPS?
    – Blaisorblade
    Jan 5, 2020 at 18:12
  • 6
    @valsaysReinstateMonica "DNSSEC does not encrypt DNS data. An observer can still look at DNS activity," Source: blog.apnic.net/2018/08/20/dnssec-and-dns-over-tls for details.
    – Nick ODell
    Jan 5, 2020 at 19:30
  • 6
    @Blaisorblade - With ESNI and dns-over-https, the hostname is never sent unencrypted. That combination isn't common now, but likely to be in future.
    – paj28
    Jan 5, 2020 at 23:31
19

Even more simple - THEIR DNS will answer with specific IP addresses of local proxies. This may not even break HTTPS - they can port forward from there. But this allows you to have the free traffic on specific IP addresses and remove those from the accounting.

Improve this answer
4
  • Wouldn't such a proxy be easily abused?
    – beppe9000
    Jan 7, 2020 at 23:10
  • 1
    Define abuse. Most end users would not even know how to handle their own DNS and the fact that providers love DNS fake blocks is well documented.
    – TomTom
    Jan 8, 2020 at 7:08
  • Abuse is when some dude packages all that in a "beautiful and easy to use" app intended for the masses :)
    – beppe9000
    Jan 8, 2020 at 12:12
  • Nope, not abuse.
    – TomTom
    Jan 8, 2020 at 13:50
4

In addition to knowing IP addresses as explained in other answers, some CDNs offer their nodes to host at ISPs. For example: Edge nodes / Google Global Cache. Data served from those nodes never leaves ISP perimeter and never hits "outbound" traffic counter. This, incidentally, is also exactly why ISP are able to offer this traffic for free or at significantly cheaper price.

Improve this answer
2

It's a mix of having a look on IP endpoints i.e ip addresses and DNS. This is usually done at the so called PGW which stands for Packe Data Gateway. The PGW is part of every mobile network.

Improve this answer
2

A small addition to the otherwise excellent answers so far. I'm aware that many ISPs, certainly all the ones I've dealt with anyway, handle this via Class of Service groups. Essentially traffic from and to specific ASNs are tagged with a specific CoS that is then considered unmetered by their stats engines. This makes the actual stat tracking easier on the infrastructure and billing easier. It also allows for throttling but not many use that today - though that might change when it comes to mobile tariffs in the future, and VBR will handle the playback flutucations this will cause.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .