FIFO buffer implementation advice

Agent Code
1

I have a cloud application that can accept late bound data, meaning that data collected by a device that can timestamp the data upon collection can be accepted by the cloud application after the fact, and populate data backwards in effect. Because of the cloud application’s ability to do this I’m looking at implementing a FIFO buffer in the device code of my Imp to buffer outgoing data that has been timestamped by the device. The buffer would really only come into play under circumstances like a WIFI outage. I’m using server.setsendtimeoutpolicy(RETURN_ON_ERROR, WAIT_TIL_SENT, 10); to keep the device code running in such a circumstance, and handling this with server.onunexpecteddisconnect(disconnectHandler); and some code to attempt reconnection to the server periodically. I’m forming up data to send to the agent like this:

// Create a data table to send to the agent
local data = {
    "impname": impName,
    "assetname": ASSET_NAME,
    "messagetype": "logevent",
    "timestamp": getDatetimeOffset(),
    "logeventtype": logEventType,
    "logeventmessage": logEventMessage
};

// Send data to agent
server.log("Sending log message to agent");
agent.send("postDataToAgent", data);

What I’d like to do is instead of sending straight to the agent, to place the data into a FIFO buffer in the device, and then have another piece of code to check the buffer for data and send to the agent if there’s anything in the buffer.

I’m after some advice as to how to implement a FIFO buffer like this in the device. Can I use a table for this? Or does anyone else have some ideas of what’s a good way to go. Just looking for a point in the right direction here.

2

How about using something like this?
https://electricimp.com/docs/libraries/hardware/spiflashlogger/

Store data on spi flash.
Read it out and send to cloud only when WiFi is working.

3

That actually looks like a great way to do it. I’m going to have a go with one of these:

4

8KB isn’t much storage. If each data entry is 80 characters, that would be 100 entries? How often is data getting written (sample rate)?

It looks like this Imp library is using i2c (instead of SPI)?

If so, that would be better because you can get an FRAM i2c breakout that is 32KB.

Maybe someone more experienced with the FRAM library could shed light on the use of i2c instead of SPI. It appears like there might be 2 libraries (for each method)? This has stirred-up my interest in maybe buying one and playing around with it. Might be useful, as it seems fairly easy to use (with the library).

5

You likely have plenty of free ram on the imp just to store it in an array of table entries… might be simplest to do that?

6

Is there any way to use the TCP transmit buffer instead? I tried increasing the size of it to 30k hoping it would act like a FIFO buffer for agent.send messages inherently but it didn’t seem to work.

7

The TCP buffer holds messages that have not been ACKed by the remote end (so TCP can retransmit them automatically, below the application layer) - during an active session. If the connection is lost or torn down, then the buffer contents are discarded.

I don’t think what you’re looking for is particularly hard to implement, though obviously memory requirements will vary depending on how many messages you want to buffer when offline.