Kinesis Firehose 500 Internal Failure Error

Libraries
1

I am trying to get the Electric IMP to send data to Amazon Kinesis Firehose. I am getting an error 500 Internal Failure each time I try to write to the temperature data to Kinesis. If I look at cloud watch logs I am not seeing any data going into Kinesis.

I am suspecting the issue is in the format of my data (see below) or an issue with my AWS setup. If I change my keys I get a authorization error so it appears my keys appear to be correct.

Has anyone go some pointers on how to troubleshoot this issue?

Thank you in advance for any assistance.
Neal

I have based my code on the following Library document:
https://electricimp.com/docs/libraries/webservices/awskinesisfirehose/

Here is my log
2016-09-05 11:00:47 UTC-4 [Agent] Agent Temperature: 77.34 degrees F
2016-09-05 11:00:47 UTC-4 [Device] Temperature: 25.19 degrees C
2016-09-05 11:00:47 UTC-4 [Agent] 500:

Here is my agent code
`#require “AWSRequestV4.class.nut:1.0.2”
#require “AWSKinesisFirehose.class.nut:1.0.1”

function reporttemp(tempCelsius)
{
//server.log(format(“Agent Temperature: %3.2f degrees C”, tempCelsius))
server.log(format(“Agent Temperature: %3.2f degrees F”, tempCelsius*1.8+32))
send_AWS_DB(tempCelsius)
}

function send_AWS_DB(tempCelsius){

const ACCESS_KEY_ID = "My Access key was removed";
const SECRET_ACCESS_KEY = "My Secret was removed";

firehose <- AWSKinesisFirehose("us-east-1", ACCESS_KEY_ID, SECRET_ACCESS_KEY);


// PutRecord
local data = {
  "temperture": tempCelsius
};


firehose.putRecord("Temperature-S3", http.jsonencode(data), function(response) {
    server.log(response.statuscode + ": " + response.body);
});

}

device.on(“reporttemp”,reporttemp);`

2

Hmm, I see this too, though some people are having luck with it. It’s not an access/signing issue (if you corrupt your secret key then you get a different error) and it’s definitely reporting an internalerror from the AWS side… but if I use the same tokens from the aws command line tool it works fine.

Looking into it…

3

I’ve worked out why this is failing; Firehose has started rejecting the content-type header “application/x-amz-json-1.0”. I’ve filed a ticket with Amazon asking why, as this seems like an API they shouldn’t be breaking.

Other people using our AWSrequest class for SQS, SNS, kinesis (not firehose), sdb, lambda, dynamoDB, etc aren’t having issues so it would appear to just be kinesis firehose that’s affected.

The workaround right now is to override the content type header in the library. You’ll need to grab the firehose class from https://github.com/electricimp/AWSKinesisFirehose/tree/v1.0.1 and paste it into your code (after the include of the requestV4 library), then modify the header setup in putRecord and putRecords from:

local headers = { "X-Amz-Target": format("%s.PutRecordBatch", TARGET_PREFIX) };

…to…

local headers = { "X-Amz-Target": format("%s.PutRecordBatch", TARGET_PREFIX) "Content-Type": "application/x-amz-json-1.1" };

…then you should see requests work fine.

I’ll update this thread when Amazon reply - this seems like a problem on their end, and given that I can’t find any documentation on exactly what the x-amz-json versions mean (they only mention 1.0 here https://docs.aws.amazon.com/apigateway/api-reference/making-http-requests/ ), it’s all very opaque.

4

Thank You very much!

Your fix worked perfectly.

5

Hugo curious if you heard back from Amazon.
Thanks again.

6

They came back and said, “hmm, yeah you’re right, and we really don’t have documentation on this stuff”. They’ve raised the issue with the product team now, but they did warn be it’d be quiet for a bit.

7

They came back and said “yeah, we need better documentation and better error messages” but that was about it. We’re updating our library to have my fix in it.