Twitter V 1.2.0 Streaming timeouts

V 1.2.1
Still getting 23 errors , but not so often. Still goes quiet for a minute or so before I get the error. Still getting the rawdata in the error message . Getting the occasional’503 (Service unavailable) from Twitter

Also , now showing an error in my error routine. Can’t find err.code

`
function onError(errors)
{
// Log all the error messages
foreach (err in errors)
{
server.error(err.code + ":-- " + err.message);
if (err.code==23) error_count++;//omly count 23 errors
}

// Close the stream then re-open it
twitter.closeStream();
device.send("reset_count",1);

twitter.stream(search_string, onTweet,onError);
}
`

2015-11-10 17:27:27 UTC+0 [Agent] ERROR: the index ‘code’ does not exist
2015-11-10 17:27:27 UTC+0 [Agent] ERROR: in onError agent_code:58
2015-11-10 17:27:27 UTC+0 [Agent] ERROR: from unknown …ricimp#twitter.class.nut#1.2.1:126

2015-11-10 17:25:46 UTC+0 [Agent] #STARWARS: 633 #MOVEMBER: 172 STRICTLY COME DANCING: 5 PASHA: 12 DAVID BECKHAM: 352 MAD MUSEUM: 0 errors 0
2015-11-10 17:27:17 UTC+0 [Agent] Stream Closed (23: count":395,“friends_count”:1177,“listed_count”:0,“favourites_count”:293,“statuses_count”:784,“created_at”:“Mon Jul 11 18:23:41 +0000 2011”,“utc_offset”:-28800,“time_zone”:“Pacific Time (US & Canada)”,“geo_enabled”:true,“lang”:“en”,“contributors_enabled”:false,“is_translator”:false,“profile_background_color”:“C0DEED”,“profile_background_image_url”:“http:\/\/pbs.twimg.com\/profile_background_images\/562096988041121793\/Sb9kqaMr.jpeg”,“profile_background_image_url_https”:“https:\/\/pbs.twimg.com\/profile_background_images\/562096988041121793\/Sb9kqaMr.jpeg”,“profile_background_tile”:false,“profile_link_color”:“4A913C”,“profile_sidebar_border_color”:“000000”,“profile_sidebar_fill_color”:“D6B57C”,“profile_text_color”:“0A0A0A”,“profile_use_background_image”:true,“profile_image_url”:“http:\/\/pbs.twimg.com\/profile_images\/663209778307424257\/X5ePKdm0_normal.png”,“profile_image_url_https”:“https:\/\/pbs.twimg.com\/profile_images\/663209778307424257\/X5ePKdm0_normal.png”,“profile_banner_url”:“https:\/\/pbs.twimg.com\/profile_banners\/333550217\/1422849920”,“default_profile”:false,“default_profile_image”:false,“following”:null,“follow_request_sent”:null,“notifications”:null},“geo”:null,“coordinates”:null,“place”:null,“contributors”:null,“is_quote_status”:false,“retweet_count”:0,“favorite_count”:0,“entities”:{“hashtags”:[{“text”:“StarWars”,“indices”:[8,17]},{“text”:“RRC”,“indices”:[38,42]},{“text”:“FNL”,“indices”:[43,47]},{“text”:“ObiWan”,“indices”:[48,55]}],“urls”:[],“user_mentions”:[],“symbols”:[],“media”:[{“id”:664131573113982976,“id_str”:“664131573113982976”,“indices”:[56,79],“media_url”:“http:\/\/pbs.twimg.com\/media\/CTd4JaOUkAAuAv1.jpg”,“media_url_https”:“https:\/\/pbs.twimg.com\/media\/CTd4JaOUkAAuAv1.jpg”,“url”:“https:\/\/t.co\/Maj99EqC1n”,“display_url”:“pic.twitter.com\/Maj99EqC1n”,“expanded_url”:“http:\/\/twitter.com\/RetroFNL\/status\/664131573491437568\/photo\/1”,“type”:“photo”,“sizes”:{“thumb”:{“w”:150,“h”:150,“resize”:“crop”},“large”:{“w”:585… (truncated)
2015-11-10 17:27:17 UTC+0 [Agent] Opening stream for: #starwars,#movember,strictly come dancing,pasha,david beckham,mad museum
2015-11-10 17:27:27 UTC+0 [Agent] Stream Closed (503: )
2015-11-10 17:27:27 UTC+0 [Agent] ERROR: the index ‘code’ does not exist
2015-11-10 17:27:27 UTC+0 [Agent] ERROR: in onError agent_code:58
2015-11-10 17:27:27 UTC+0 [Agent] ERROR: from unknown …ricimp#twitter.class.nut#1.2.1:126

Not sure why the code value is not being relayed to your error handler after a 503; the source code suggests it does.

I’ve tweaked 1.2.1 a little, and adopted your onError() code to see if I get the same.

Spotted the reason for the missing ‘code’ - should be fixed now

One other factor that might be influencing this is the limitation on agent HTTP downloads. This is currently 512Kbytes, and applies to all HTTP connections, even streaming ones. After your 512Kbytes are used up, the connection will fail with error 23.

Peter

Have we got any ideas and a timescale for fixing this ? I’ve put my application on hold for the time being as the delay before the stream is restarted after an error now seems to be 3-5 minutes which make it very unresponsive. Is this delay down to Twitter or the library code ? If its Twitter presumably we can’t do anything about it ?

Also my local error routine failed again (same as before) couldn’t find err.code. or err.message I’ve commented out the line for the moment and I need to test it again to be sure.

Apart from the delays after an error everything else seems very stable.

Regards

John

`
function onError(errors)
{
// Log all the error messages
foreach (err in errors) {
// server.error(“My error” + err.code + “: " + err.message);
server.log(” Some Error");
}

// Close the stream then re-open it
twitter.closeStream();
device.send("reset_count",1);

twitter.stream(search_string, onTweet,onError);
}
`

It sounds like you have enough traffic there to cross the 512kB limit which, to be honest, should only be enforced on each chunk when streaming. I’ve filed a bug about it and am waiting for an opinion on whether it’s a hard fix or not.

I’ve updated Twitter 1.2.1, @wheable. It seems more resilient to ‘Twitter busy’ errors for me, but let me know how you get on.

Thanks, @smittytone I’ll give it a try

Hi again,
Still happening , went unresponsive for 6 mins :frowning:
It doesn’t seem to be related to the number of 'hits’
I’ll leave it to stew a bit.
I’m assuming I only have to reference Twitter 1.2.1, re-compile and download to pick up the revised library ?

John

2015-11-17 15:25:58 UTC+0 [Agent] JAMES MARTIN: 0 #MOVEMBER: 103 STRICTLY COME DANCING: 22 PASHA: 14 DAVID BECKHAM: 182 MAD MUSEUM: 0 errors 0 2015-11-17 15:31:44 UTC+0 [Agent] Opening stream for: james martin,#movember,strictly come dancing,pasha,david beckham,mad museum 2015-11-17 15:31:52 UTC+0 [Agent] JAMES MARTIN: 0 #MOVEMBER: 103 STRICTLY COME DANCING: 22 PASHA: 15 DAVID BECKHAM: 182 MAD MUSEUM: 0 errors 0

You don’t even need to change your code - just click ‘Build and Run’.

I’m Still getting regular breaks in the flow of Tweets from the library/Twitter feed @smittytone. Are you seeing this ? I can put up with the occasional ‘Messages Throttled’ and ‘503’ errors but these regular ‘dead’ periods are a real application killer :frowning:

They come every minute or so and last between 30 seconds and 2 minutes.
I’m still not sure if this is a problem with Twitter or the library code . I need to know if its worth pursuing . If its a basic limitation in the Twitter stream then presumably we can’t do much about it ?

I can give you my imp ID/agent link if this helps.

According to Twitter, “Clients which do not implement backoff and attempt to reconnect as often as possible will have their connections rate limited for a small number of minutes.” This sounds like what is hitting you.

As it stands the library, if it receives a ‘you’ve been rate limited’ response, it tries again in 60 seconds, but there’s nothing to stop Twitter responding immediately with another ‘you’ve been rate limited’. The library doubles the connection-retry period each time. Previously, it doesn’t eventually reset the retry period; I have now changed this in the 1.2.1 test version you’re using, and added a log post to see if it is rate-limiting that’s triggering your pauses.

@smittytone Thanks, I get this syntax error
Syntax error (line 6): newline in a constant in file ‘twitter.class.nut:1.2.1’ row 129 col 57

I’ll go and have some lunch and leave you in peace for a bit :slight_smile:

Missing double-quote! Sorted now.

Fine now :slight_smile: I’ll give it a go

If its of any help , here’s a record of the pauses over the last hour.
Seems to have calmed down a little maybe

`
Elapsed 0 D 0H 4m 16s Waited 46 secs
Elapsed 0 D 0H 6m 5s Waited 80 secs
Elapsed 0 D 0H 10m 18s Waited 72 secs
Elapsed 0 D 0H 12m 8s Waited 60 secs
Elapsed 0 D 0H 14m 35s Waited 81 secs
Elapsed 0 D 0H 19m 47s Waited 57 secs
Elapsed 0 D 0H 20m 37s Waited 34 secs
Elapsed 0 D 0H 21m 24s Waited 19 secs
Elapsed 0 D 0H 23m 36s Waited 42 secs
Elapsed 0 D 0H 24m 20s Waited 33 secs
Elapsed 0 D 0H 25m 4s Waited 30 secs
Elapsed 0 D 0H 28m 25s Waited 45 secs
Elapsed 0 D 0H 29m 57s Waited 41 secs
Elapsed 0 D 0H 31m 9s Waited 40 secs
Elapsed 0 D 0H 32m 46s Waited 44 secs
Elapsed 0 D 0H 33m 47s Waited 51 secs
Elapsed 0 D 0H 36m 40s Waited 54 secs
Elapsed 0 D 0H 37m 50s Waited 54 secs
Elapsed 0 D 0H 40m 25s Waited 44 secs
Elapsed 0 D 0H 42m 42s Waited 45 secs
Elapsed 0 D 0H 45m 29s Waited 42 secs
Elapsed 0 D 0H 46m 52s Waited 46 secs
Elapsed 0 D 0H 48m 10s Waited 52 secs

`

Seems to be getting worse , a couple of ten minute downtimes :frowning:
@smittytone . Does your log show any rate-limiting ? Any ideas ?

@smittytone I’ve had a thought.
Could the ‘23’ errors relate to the agent-device communications ? My agent code is getting a tweet and immediately sending a notification to the imp. So the agent-imp comms have to go at least as fast as the twitter-agent connection or (presumably) messages will be buffered ? if the buffer fills up then we get an error ?
I’ve modified my code to simply count the relavent tweets and then send an updated count at 1 second intervals . So far no errors :slight_smile:
Let me know if I’m talking nonsense.

@smittytone
My application has been running for an hour with not a single error !!! I think we might have cracked it !! I’ll get paid and The kids will get Christmas presents after all :slight_smile:
Thanks for your help with this.