Dropping data samples to Grovestreams

I had a program operating sucessfully using Imp and Grovestreams, made a few changes, then it wouldn’t function properly.
After a lot of messing around, I decided to revert to the example given here, which used to work reliably.

https://www.grovestreams.com/developers/getting_started_elec_imp_temp.html

Sometimes 3 or 4 samples in a row make it, sometimes samples are lost for 1 to 10 sample intervals.
The agent code looks for 200 being returned as a success, and if other than that logs the error code to server, but returns are always blank when data is not transmitted sucessfully!!.
I already discussed this with Grovestreams, they said all was ok at their end…any ideas??

OK, I just tested the Plotly equivalent example…
https://plot.ly/electric-imp/tmp36-temperature-tutorial/

…and it delivers data flawlessly, so I don’t believe its an Imp issue, or anything with my hardware, but maybe something in the recent Imp update affected Grovestreams compatibility??

What error codes is grovestreams returning?

Could be they are rate limiting from our server IP addresses…

There are documented ways to circumvent the rate limiting which by default is one push every 10 seconds (per component, I believe) – if that’s your issue. How often are you pushing data? Actually, they do return an error code if you’re exceeding that limit.

I push lots of data, usually once per minute from many sites and have not experienced any data loss. Mike Mills, the founder of GroveStreams, has been extremely helpful in providing assistance when suspected issues arise. Along with great functionality and value, this is one of the reasons I don’t hesitate to recommend them. Have you posted on their forum?

Suggestion – you should turn on “API tracing” in your GroveStreams account and have it log for a sufficient number of minutes. That will show you exactly what they’re receiving.

Thanks guys…

API monitor reveals successful messages, no bad ones.
Return in agent server log shows no returned error codes.

GS has been very reliable until this week, problems coincident with code changes I made, so I initially thought I was the problem, and I may still be!!

Then tried GS thermocouple example which revealed the same issues.
Don’t believe it is a rate limiting issue, as it was failing with 1 minute data push in my own code, GS’s own thermocouple example has 20 sec data…several successive 20 sec logs would make it sometimes, then nothing for a few samples, or for a couple of minutes.

I originally posted this issue on their forum but was getting nowhere myself, so decided to try Plotly as a sanity check.

I now have given my GS API code to Mike at GS and they are investigating further.

It turns out that, for some reason, sending URL to GS no longer works in https:// form, but as http:// it works just fine, as suggested by Mike M at GS.

https:// usage is shown in the GS Thermocouple example, and it used to be flawless that way.

Is it possible that Imp updates have resulted in a more pedantic security methodology that somehow before GS was managing to skirt around??

From what I’ve read above, it seems like people are getting no error codes returned - which implies that the HTTPS transaction was fine (error numbers <100 indicate problems with the mechanics of the connection vs a code returned from the remote server). If their HTTPS server used an encryption type that we do not support because it’s deprecated from a security point of view, for example, you’d see an error code.

That tends to imply that the breakage could be in grovestream’s https servers that are accepting data and then throwing it away?

We (GroveStreams) have reproduced it. We don’t see any errors in any of our web logs related to this. The Imp response.statuscode is returning an error code of 56. Is there a specific http client library that is being used by the Imp agent? Maybe CURL? Maybe that library is returning the code?

We did just upgrade our SSL certificates lately. The issue is probably related to that change.

Ah, so there are errors - other users had not noted any; yes, we use libcurl so error 56 is CURLE_RECV_ERROR. We’ll look into what that means, it sounds like we’re not getting something back from your end but that’s just an assumption based on the name of the error :slight_smile:

I just checked again using the GS Thermocouple example…I am honestly not seeing any any returned error code, just blank ???

if(response.statuscode != 200)
{
server.log("error sending message: " + response.body);
return null;
}

The error number is in the statuscode:

`   
if(response.statuscode != 200) {
   server.log("error sending message. Status Code: " + response.statuscode + ". Response Body: " + response.body);
   return null;
}
`

Thanks…I didn’t know how that was being seen

We cannot reproduce it using CURL from the command line.

When we upgraded our SSL certificates we also disabled SSLv2 and SSLv3.

When an Imp Agent runs, does it always run on the same server each time or can it run on different servers? I’m wondering if some Imp servers are configured to use SSLv2 and SSLv3 or maybe they’re running an older version of CURL. That would explain the intermittent part of the error. Just a guess…

We don’t accept SSLv2/v3 so it shouldn’t be that…

Agents can run on many servers, and their IPs will change; generally developer devices tend to be on a single server. All have identical configurations, though (every server is running identical code).

We’ll try to replicate here, though, and if we can catch it often enough get a pcap of the failure. If the issue is in the TLS negotiation then that’s debuggable without your server key, otherwise you’ll have to look at the pcap.

We’ve changed our Imp sample code to use http for now (instead of https) until we get this sorted out.

Been debugging what appears to be a similar issue for another customer; what I saw was that the server end was rejecting our SNI option in the TLS client hello with a warning about unrecognized server name, and that was causing us to drop the connection.

Does your server have the SNI set correctly? See https://en.wikipedia.org/wiki/Server_Name_Indication and check for ServerName / ServerAlias lines in your apache config (I see apache is being used for at least some of the grovestreams traffic).

I’ve confirmed this is happening; using curl 7.22.0 if I curl to https://grovestreams.com (whilst tcpdumping the connection) I can see the grovestreams.com server saying the hostname is unrecognized - ie SNI is not set correctly on your Apache.

Though some people ignore these errors, we’re strict on them…

Screenshot below.

That was it. We needed to change our Apache ServerAlias settings. Our Imp seems to be working fine now. Thanks Hugo!