If I understand correctly, the operating system gets subnet mask together with IP and other stuff from the DHCP server. And this subnet mask seems to be important for router because it knows if packet goes from/to LAN or should be sent somewhere else or it's received from Internet. But why is the subnet mask needed for the operating system? What does the operating system do with the knowledge of the subnet mask?
There are 3 major components to the network configuration:
- IP Address
- Network Mask
- Default Gateway
The Operating System needs to know the network mask in order to determine where the data needs to travel. Your IP address combined with the network mask tell the OS whether a target IP address exists on the local subnet, or if it exists outside of your subnet. Some computers have multiple network connections, whether physical (multi NIC) or logical (think VPN). Without a subnet mask for each network, the OS would not be able to determine which traffic should leave which interface.
If the target exists within your subnet:
- Your OS determines the hardware address of the destination by referencing its ARP table. If there is no entry in the ARP table for that address, the system sends out a hardware broadcast that is flooded to all devices on the layer 2 network. The system who owns the IP address would then respond to the broadcast with its hardware address (MAC ADDR). The OS would then send the data along, with it never leaving the local network.
If the target does NOT exist within your subnet:
- Your OS determines where to send the packet based on its routing table. Most often than not, the packet will be destined for the default gateway. This gateway is generally a router which will use its own routing tables to send the packet on its way to its final destination. The process of sending to the default gateway is the same as sending to another host on the subnet. If the gateways MAC address isn't in the ARP table, it will send a broadcast out to determine the gateways hardware address, and then forward the frame.
So as you can see, the subnet mask is very important as it is used to determine the location of the destination network, and route them accordingly.
Based on the combination of an IP and a subnet mask the OS can figure out if a destination IP that it's user space application wants to contact is in it's local network or needs to be sent via some router. If it needs to be sent via a router, it does a routing table lookup to determine which router to use.
See Mauro's answer, but in addition the subnet mask allows the host to determine the IP broadcast address. I think this is less important in these days of DNS based name resolution, but in the old days before Active Directory NETBIOS name lookup could fail if the subnet mask was wrong because the broadcast address would be wrong.