Best practises for large data transfers

Squirrel Code
1

Are there any best practises for performing large data transfers Agent <-> Device.

Could you just pass a 1MB blob to MessageManager.send()? Or are you best to break it up into smaller chunks? Is there an optimum chunk size?

Thanks

2

Even on imp005 – the only extant imp type which stands a chance of allocating a 1MB blob – you’d likely soon get in a pickle due to heap fragmentation, a situation where there might be more than 1MB free altogether, but not in a single continuous chunk.

So for that reason alone, it’s best to break things into smaller chunks to help the device-side cope. To keep things running smoothly, the chunk size in the device-to-agent direction should be no bigger than the TCP output window set by imp.setsendbuffersize().

A smallish chunk size would also improve reliability for imps at the limits of their wifi range: even if the connection keeps dropping, you can still make (gradual) progress if you can get a chunk at a time across the connection.

Peter

3

Also, the new mode RETURN_ON_ERROR_NO_DISCONNECT is great for streaming; you can keep the TCP buffers full whilst still maintaining squirrel responsiveness to deal with other things during the send.

Some example code here: https://gist.github.com/hfiennes/56b9c359479b7d13245956f3d573f357

You can experiment with chunk sizes; I found that 4k or 8k seemed optimal for maximum throughput.