Best practises for large data transfers


Are there any best practises for performing large data transfers Agent <-> Device.

Could you just pass a 1MB blob to MessageManager.send()? Or are you best to break it up into smaller chunks? Is there an optimum chunk size?



Even on imp005 – the only extant imp type which stands a chance of allocating a 1MB blob – you’d likely soon get in a pickle due to heap fragmentation, a situation where there might be more than 1MB free altogether, but not in a single continuous chunk.

So for that reason alone, it’s best to break things into smaller chunks to help the device-side cope. To keep things running smoothly, the chunk size in the device-to-agent direction should be no bigger than the TCP output window set by imp.setsendbuffersize().

A smallish chunk size would also improve reliability for imps at the limits of their wifi range: even if the connection keeps dropping, you can still make (gradual) progress if you can get a chunk at a time across the connection.



Also, the new mode RETURN_ON_ERROR_NO_DISCONNECT is great for streaming; you can keep the TCP buffers full whilst still maintaining squirrel responsiveness to deal with other things during the send.

Some example code here:

You can experiment with chunk sizes; I found that 4k or 8k seemed optimal for maximum throughput.