Strategies for compressing data useful? device - agent - IDE log or browser

Sorry in advance for my noob question. I am not sure if I have even enough knowledge to phrase this correctly. My device and software are working fine but I would like to work on optimizing the code and improving the responsiveness.

I have a couple of watt meters using ADE7953 (same as Becky dev board). For the most part I will send out small amounts of data but I also have a function to capture waveform snapshots and this creates a rather large JSON array of 60 pairs of values for current and the same for voltage.

JSON is nice for the human readability but I looked at this and wondered if I should be trying to leave the data in binary.

On a slighty related issue, I have been converting the register counts from the ADE7953 and converting them to real physical values such as “7.328” volts. I only just recently considered the idea I could send the 3-byte register value to the agent and in the agent perform the conversions and from the agent log to the IDE (or send to the web page).

My question is one of strategy but I will try to pare it down to this:

- Is there some efficiency in passing squirrel tables between the device and the agent?
- if this is the case I think it is better to pass a table of integer values from the device to the agent and scale them from there converting them to decimals and logging. Maybe the imp service employs its own compression?

  • Is there a way to and is it advisable to use binary in JSON to cut down on the size of the data passing to the web browser?

one reason this occurred to me is that I sometimes notice significant delays in getting the data to my web app.


for illustration: when I read volts from the ADE7953 it is a 3-byte value such as

Decimal: 140579 counts
Binary: 100010010100100011 (which of course is just 3 bytes)

After conversion to volts and displayed in the IDE log it is “7.324,V”

thanks for the reply!

ya, I thought more about this after I banged out that message on my lunch hour. The bottleneck on some client devices such as my Galaxy S might be rendering the highcharts chart. I probably could time that with some code in my web app but this is not something I have attempted before.

I would not plan to use the IDE log as a normal use model but I am still curious about the efficiency of using it.

If I log from the device I think the data might flow like this:

device - imp service - web browser/IDE log

if I send data from the device to the agent

device - imp service? - agent? - imp service? web browser/IDE log.

Seems like a longer trip to log from the agent but on the other hand the dataset might be smaller out of the device.

I don’t think the device converts any JSON so there isn’t anything to time. I am curious how the data gets from the device to the agent. I am just sending a squirrel table so I don’t see what format it is in.

If I were extremely limited in bandwidth from the imp device then I would wish to push data from it in as small a format as possible (maybe squirrel tables instead of planner log strings)

am I making any sense?

mjkuw94, have you tried timing the formatting of all 60 of your values and their conversion to JSON? That could also take a long time, as Squirrel isn’t so efficient at converting binary stuff into strings. Using hardware.millis() at the beginning and end of your conversion routines, you can work out how long it’s taking by posting the difference between the values to server.log(). I wouldn’t consider compressing the data you transfer until you know that it is indeed the bottleneck. I do use JSON to send some data between device and agent, but I don’t bother with binary blobs. Instead, I do all the conversion and formatting in the agent.

Well, if you’re planning on using your device commercially, there may be limits on throughput, rather than bandwidth. That is likely to be influenced more by how often you transfer data rather than the size of the payload. The IDE actuallys tells you at the (end of a connection) how much data you transferred between device and agent. This can be a very useful metric.

no, it is not commercial. I try to not be greedy with the data and the occasional snapshot that I take is so rare I doubt it adds up to much.

Do you mean the commercial IDE states how much data has transferred?

No, in your log a message will be posted a couple of minutes after your imp is disconnected.
2014-02-25 17:54:24 UTC+23: [Status] Device disconnected; 4144 bytes sent, 912 received, 5056 total

Frustratingly, this only appears in certain circumstances, but is useful nonetheless. Try using your imp for a few minutes, then shutting it off and waiting.

in order to read the waveform, I first read i2c data into a blob. only later, just after completing the acquisition is it converted into an array.

Perhaps I should send the blob from the device to the agent. Is that significantly more efficient in terms of getting data over wifi and out of the imp device?

Sending the blob from device to agent will minimize the amount of data traffic. It will really have no affect on speed; the wifi is running at a minimum of 1Mbps and likely 20Mbps+, and the processor is fast enough that this is not an issue.

Thanks Hugo, it is what I was guessing. I’ll be working on this shortly. I don’t have a great way to measure that difference so it is helpful to have you giving the real information.

Really the best way is via the status output in the log. If you, say, do 100 readings then use server.disconnect(), then you can see how much data was taken to send these 100… then try different methods.