I'm creating a server on an embedded system. I have a FAT filesystem on an SD card which contains a file, which should be sent via HTTP to some client over wifi that connects to the system.
At first I tried to read the file byte by byte, which was very slow. To improve the file reading, I introduced a buffer. I read from the file to the buffer and then write the buffer content to the stream that is sent to the client:
file -> buffer -> network
Increasing the buffer size increases the data transfer rate. I assume this is because of the reduced number of read operations to the file system.
However, an arbitrary large buffer (say 5000 bytes) is counterproductive. I suspect this is due to bufferbloat. The server spends too much time filling the buffer, which confuses the network.
Apparently, there's some optimal buffer size, where read speed is increased without causing a slowdown due to buffer bloat.
I would like to increase the overal performance in terms of data transfer rate. Currently I get like 3Mbi/s which is too slow. I am in control of the buffer size and the cluster size. Can I improve the performance of the system by making good choices for those two values?
- Should buffer size and cluster size ideally be the same? Should buffer size be some integer multiple of the cluster size in general (possibly >1)?
- Should I choose the two in some way that they also line up with the network payload size? (I'm not much experienced with networks and don't know if there even is such a thing as a fixed payload size)
How do I decide on those sizes to increase data transfer rate?