I'm creating a server on an embedded system. I have a FAT filesystem on an SD card which contains a file, which should be sent via HTTP to some client over wifi that connects to the system.

At first I tried to read the file byte by byte, which was very slow. To improve the file reading, I introduced a buffer. I read from the file to the buffer and then write the buffer content to the stream that is sent to the client:

file -> buffer -> network
  • Increasing the buffer size increases the data transfer rate. I assume this is because of the reduced number of read operations to the file system.

  • However, an arbitrary large buffer (say 5000 bytes) is counterproductive. I suspect this is due to bufferbloat. The server spends too much time filling the buffer, which confuses the network.

Apparently, there's some optimal buffer size, where read speed is increased without causing a slowdown due to buffer bloat.

I would like to increase the overal performance in terms of data transfer rate. Currently I get like 3Mbi/s which is too slow. I am in control of the buffer size and the cluster size. Can I improve the performance of the system by making good choices for those two values?

  1. Should buffer size and cluster size ideally be the same? Should buffer size be some integer multiple of the cluster size in general (possibly >1)?
  2. Should I choose the two in some way that they also line up with the network payload size? (I'm not much experienced with networks and don't know if there even is such a thing as a fixed payload size)

How do I decide on those sizes to increase data transfer rate?

  • Any tuning you do is essentially pointless as it's the SD card that is highly likely to be the lowest performing part of the IO chain. – Chopper3 Nov 1 '16 at 9:12
  • @Chopper3 going from single byte read/write to a buffer of 300 byte increased the speed significantly, which suggests that tweaking the sizes has an effect. I agree that eventually the card will be the bottleneck and I'd love to have that problem, however, at the moment, the issues appear to be elsewhere. Point in case: the class 4 card supposedly does 4MB/s which I am very far away from at the moment with 3Mbit/s. – cross Nov 1 '16 at 9:24
  • What the hell kind of filesystem only pulls files byte-by-byte? or are you writing this code yourself? – Chopper3 Nov 1 '16 at 9:29
  • @Chopper3 I am indeed writing this code myself. – cross Nov 1 '16 at 9:32
  • 1
    Ah, then you didn't read the text we put in the login page about what we are and aren't for - I think you probably want our sister site stackoverflow.com. – Chopper3 Nov 1 '16 at 9:51

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.