Optimizing download buffers


#1

I am trying to download large files to an Imp at the same time as processing other data, and I have very little free memory to play with.

I am downloading files in 5kb chunks by using device.send() and agent.on(). I want to download as fast as possible, and I’ve noticed that the vast majority of the download time is spent waiting for messages to travel between the device and agent, so I’m hoping to increase the size of my download buffer. Currently, after some time downloading, I am crashing with an out of memory error. My most recent logs before that indicate that I have around 10kb free memory, so I think that the problem is that I don’t have 5kb of contiguous memory to allocate to the blob as it comes into the callback for agent.on().

I have a few questions about this:

  1. Can you tell me any more about how the memory allocation works for data sent through device.send() to agent.on()? I imagine that the OS loads the data into some sort of network buffer, and then attempts to copy this over into a squirrel object.

  2. Is there any way for me manually allocate the space for the data that the agent.on() callback receives? So if I allocate a 5kb buffer at boot, and I know the data received will be 5kb, is there any way I can make sure that the data goes directly into the existing buffer without allocating more memory?

  3. When I call device.send(data), data is a 5kb blob. Are there any potential pros and cons to making data an array of multiple smaller blobs? The idea would be that, e.g., if data was an array of 5 1kb blobs, then the device could have an easier time allocating memory for 5 1kb blobs then for 1 5kb blob, due to contiguity issues.

Any other tips for increasing download speed on the Imp would be appreciated,

Thanks!