17

I am trying to understand application layer protocols in TCP/IP stack. I know that both HTTP and DNS protocol stay at the top layer (Application Layer). So, when a browser wants to access a resource, it has to send a request to the HTTP server, as for example:

GET www.pippo.it/hello.htm HTTP/1.1

Making this request following the rules of HTTP protocol, it uses the page URL, not the IP address.

I know that DNS request is necessary to convert URL to IP. So my question is: does HTTP invoke the DNS protocol? It seems impossible to me, since both are top layer protocols (so DNS can't provide a service to HTTP). In the same way even TCP (which stays on a lower level) can't ask for a service at a higher level protocol like DNS.

So when does the DNS request happen? And who performs such a request?

  • 1
    Could you accept one of the answers to clarify which of these answers the question? – 030 Nov 16 '14 at 15:57
38

The HTTP request in question is actually not valid unless the browser is talking to an intermediary (proxy).

Your example would look a bit more like the following if the browser was talking with a web server directly:

GET /hello.htm HTTP/1.1
Host: www.pippo.it

Now, to put this in perspective, consider the OSI model:

The OSI model

We have 3 systems in action:

  • A client running the browser
  • A web server serving the site
  • A DNS server knowing the IP address of the site

The protocols involved are, bottom to top (minimum relevant set to OP):

  • IP
  • TCP, UDP
  • HTTP, DNS

The HTTP communication is done over the TCP protocol (TCP is on top of the IP protocol) while the DNS communication, in this case, is done over the UDP protocol (UDP is also on top of the IP protocol).

Here's the communication sequence in short:

  1. The client, running the browser, asks the DNS server for an A record for www.pippo.it, using the UDP protocol.

    1.1. On the client, it's the operating system that does the resolving part and talks back to the browser --- the browser never talks to the DNS server directly, rather through the OS by invoking gethostbyname() or the newer getaddrinfo(). On Windows, the order in which the OS resolves addresses is likely defined by something like this, while on Linux the resolving precedence is defined by /etc/nsswitch.conf

  2. The DNS server, using the UDP protocol, responds to the client with a record/IP address, if it exists

  3. The client opens a TCP connection on the port 80 of the web server and writes the following text:

    HTTP request:

    GET /hello.htm HTTP/1.1
    Host: www.pippo.it
    

    You could mimic the same thing by doing something like this in your console or command prompt:

    > telnet www.pippo.it 80
    Trying 195.128.235.49...
    Connected to www.pippo.it.
    Escape character is '^]'.
    GET /hello.htm HTTP/1.1
    Host: www.pippo.it
    

    followed by two empty lines. If the requested content exists, the web server will print it on the screen. If there's a browser on the other side, the response text gets parsed by the browser, and all tags, links, scripts and images are rendered in what we call a web page.

In reality there are some more details, e.g. browsers may cache IP addresses if you already visited some domain, so that DNS resolving becomes unnecessary. Also, modern browsers may try to do the resolving before you actually need it (DNS prefetching) to speed up your browsing.

Additionally, your computer may have static records in a hosts file. If a record matches the request, the local static entry gets used first and no DNS server is ever contacted. This is configurable, and not necessarily true, but it's the default on the operating systems I'm familiar with.

  • 4
    @trikly: that's what the 'Host' header is for. Without it you can only have one website per IP address. This is a key difference between HTTP/1.0 and HTTP/1.1. Thankfully HTTP/1.0 browsers are rare now - but if you want to cater for them then you need a different IP address for each site (they can still be hosted on the same server). – A E Nov 12 '14 at 11:06
  • 1
    @AE Thanks. I think I was unclear in my question, and that's why Hrvoje didn't understand what I was saying. (I should have said domain rather than URL). I'm glad you still understood. – trlkly Nov 12 '14 at 11:08
  • 1
    You say "This is not correct HTTP request", and that's mostly true, but it's closer than you imply: "To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI form in requests". (RFC 2616 §5.1.2) So GET http://www.pippo.it/hello.htm HTTP/1.1 would be a valid, if unusual, request. It would also be a valid and usual request to an HTTP proxy. – wfaulk Nov 12 '14 at 22:44
  • 1
    gethostbyname() is somewhat outdated. One would better use getaddrinfo()... – glglgl Dec 10 '14 at 21:14
  • 1
    @Utku unfortunately no because SSH presumes the other end speaks SSH protocol while telnet is plain text protocol and can be used to talk to any other plain protocol like say POP3, IMAP as long as they don't use SSL/TLS in which case you would need to wrap telnet session with helper like sslwrap or something similar. – Hrvoje Špoljar May 12 '16 at 22:05
12

HTTP is transported over TCP, which is a IP protocol. To make an HTTP request, the browser has to open a TCP connection, and do to that, it needs the destination IP address (i.e. the IP address of the server). To resolve the server's hostname, it has thus to issue a DNS request (generally the DNS request itself is sent by the operating system when a program calls its name resolution functions; however, nothing prevents a program from sending DNS requests by itself to the DNS server). Once the connection is established, it can send its HTTP request, which contains the path to the requested resource, and a Host field with the hostname of the server (e.g., Host: www.pippo.it). The hostname does not go on the request line (it would actually be GET /hello.htm HTTP/1.1), except when the request is sent to an HTTP proxy (and in this case, the full URL is present, including the protocol part, e.g. GET http://www.pippo.it/hello.htm HTTP/1.1),

  • Thanks, now it is clearer, but not completely. You write that browser has to issue a DNS request. Ok, but after having received the IP from DNS server, how does it use it? I mean, such IP doesn't appear in the HTTP request. So I assume there is still one more step before issuing the HTTP request and I think it is the opening of the connection. This point is not very clear to me... Thanks once more! – Giancarlo Perlo Nov 11 '14 at 13:59
  • 5
    Indeed, the IP is needed to open the TCP connection, inside which the HTTP request is transported. Actually, the IP address of both client and server is sent along with ALL packets of the connection. The best way to learn how this works is probably to install a packet capture tool (Wireshark is an excellent multi-platform and open source one), capture a simple HTTP request, filter it out from the rest of network activity, and look at how all the packets were sent on the wire. You should actually be able to see the DNS request before the TCP connection, too. – Ale Nov 11 '14 at 14:47
  • 1
    Proxied requests should be using the Host header, not putting the full URL in the GET line. – Stop Harming Monica Nov 12 '14 at 11:32
  • 1
    @OrangeDog: No, on the contrary. RFC 7230 (section 5.3.2) explicitly says that a client making a request to a proxy MUST use an absolute-URI in the request line. (There must still be a Host header duplicating information from the request line; section 5.4). – Henning Makholm Nov 12 '14 at 16:31
7

The procedure goes like this:

  1. The user (you) gives the browser a URL, like http://www.pippo.it/hello.htm
  2. The browser splits that into three parts:

    • Protocol http
    • Hostname www.pippo.it
    • URL path /hello.htm

    (a more complicated URL could have other parts too, I'll ignore that possibility for now)

  3. The browser knows that in order to create an IP connection, it needs an IP address. To obtain an IP address, it needs to use DNS (unless it has the address cached).

    1. The browser asks the operating system for the IP address of a DNS server; suppose it gets 8.8.8.8.
    2. The browser constructs the following multi-layered connection:

      • IP layer: connect to 8.8.8.8
      • UDP layer: set packet for destination port 53
      • DNS layer: create a DNS request for an A record for the hostname www.pippo.it

      Of course I'm omitting a lot of detail about e.g. the exact format of the packets involved.

    3. The browser receives a DNS response (layered on top of UDP layered on top of IP etc.) which gives the IP address for www.pippo.it, let's say it's 10.11.12.13
  4. The browser knows that in order to create a TCP connection, it needs a port number. To obtain a port number, it looks up the protocol http in its internal table and learns that it should use port 80.
  5. The browser constructs the following multi-layered connection:

    • IP layer: connect to 10.11.12.13
    • TCP layer: set packets to destination port 80
    • HTTP layer: create an HTTP request for the URL /hello.htm on the host www.pippo.it (because the computer at 10.11.12.13 might be hosting several domains, so it needs to know which one is desired)

      GET /hello.htm HTTP/1.1
      Host: www.pippo.it
      ...
      

    Of course I'm omitting all the details of the TCP handshake and such.

  6. The browser receives an HTTP response (layered on top of TCP layered on top of IP etc.) containing the contents of hello.htm

And for good measure, I'll mention that the browser now examines the content of that response, and identifies any additional resources needed: images, CSS, Javascript, etc. Then it repeats this whole process for each such resource.

  • 4
    Step 3 really is not something that the application itself does. The application just uses something like getaddrinfo or gethostbyname to ask the OS to resolve the address for it. Also, the OS typically uses multiple mechanisms to try to look up names, not only DNS. (Typically at least the hosts file in addition to DNS.) – Håkan Lindqvist Nov 11 '14 at 17:28
  • Thanks! It is a really impressive and detailed answer and a very useful too! – Giancarlo Perlo Nov 11 '14 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.