Imp Stuck in "firmware update triggered"

And if the length or the bytes are wrong, try again with an HTTP “Pragma: no-cache” header – which, if you’re using wget, is the “–cache=off” option…

Peter

Hi Peter. This is the result from wget:

adam-macbook_pro:Downloads adam$ wget http://upgrades.electricimp.com/upgrades/0002d7000001.2.dld
–2014-09-21 06:49:53-- http://upgrades.electricimp.com/upgrades/0002d7000001.2.dld
Resolving upgrades.electricimp.com… 54.183.22.232
Connecting to upgrades.electricimp.com|54.183.22.232|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 622172 (608K) [application/octet-stream]
Saving to: ‘0002d7000001.2.dld’

100%[======================================>] 622,172 690KB/s in 0.9s

The bytes in the content are as you described.

And these are the headers:
HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sat, 20 Sep 2014 18:59:39 GMT
Content-Type: application/octet-stream
Last-Modified: Fri, 05 Sep 2014 16:27:44 GMT
Accept-Ranges: bytes
Content-Length: 622172
Connection: Keep-Alive
Age: 125

Looks like it all works that way - of course, its not simulating the double-download that the Imp does…

What should I test next?

Well that all looks completely shipshape. It’s reordered the headers and added one for “Age”, but that should be unimportant.

I suppose you could try the same thing but, instead of “0002d7000001.2.dld”, use (your-imp’s-MAC-address-no-colons-lower-case)+".2.dld". So it’d be (something like) http://upgrades.electricimp.com/upgrades/0c2a69xxyyzz.2.dld but with the last six characters of your MAC address instead of the “xxyyzz”.

This one won’t be the same file as before: it should be a 560,880-byte file which starts with 0x49 0xAB 0xCE 0xC3 and ends with 0xCE 0x0E 0x5F 0x91.

Have you now upgraded all your imps at the office, or have you one left you could try upgrading at home? Maybe Vodafone’s proxy server just went bananas for a bit and now they’ve fixed it.

Peter

Hi Peter,

I have tried the new test - and I get a problem… my download result is only 145000 bytes… ( I have masked my MAC from this forum post, but did indeed send it correctly)

Interesting that the result headers say “last modified in 2012” - that certainly looks hinky.

–2014-09-21 16:56:58-- http://upgrades.electricimp.com/upgrades/0c2a69XXXXXX.dld
Resolving upgrades.electricimp.com… 54.183.22.232
Connecting to upgrades.electricimp.com|54.183.22.232|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 145000 (142K) [application/octet-stream]
Saving to: ‘0c2a69XXXXXX.dld’

100%[===================================================================>] 145,000 89.6KB/s in 1.6s

2014-09-21 16:57:01 (89.6 KB/s) - ‘0c2a69XXXXXX.dld’ saved [145000/145000]

it also doesnt start or end with anything like what you have described.

These were the headers returned:
HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sun, 21 Sep 2014 05:02:21 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 30 Jul 2012 21:38:28 GMT
Accept-Ranges: bytes
Content-Length: 145000
Connection: Keep-Alive
Age: 0

And run again with the --cache=off, comes back again with the identical 140000 bytes, and these headers
HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sun, 21 Sep 2014 05:04:48 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 30 Jul 2012 21:38:28 GMT
Accept-Ranges: bytes
Content-Length: 145000
Connection: Keep-Alive
Age: 0

The suffix to your mac address needs to be ‘.2.dld’ in that last test.

Ah - good spotting Philmy. Yes that worked correctly - 560880 bytes with the correct beginning and trailing byte sequences.

Back to square one…

Note: I do have 3 imps - 2 I upgraded successfully through a different ISP - but I left the last one un-upgraded to further our diagnostics - so if there is something I can do with that on my Vodafone connection, let’s have a shot at that.

Thanks

One thing that might scupper upgrades, would be if the caching server spotted that two URLs had the same content and issued 30x redirects instead.

On the Vodafone network, could you “wget -d” the upgrade for your imp’s MAC address (as before) to make sure it’s in their cache, then “wget -d” the upgrade for your-MAC-address-plus-one (i.e., as if a different imp also asked to be upgraded), then “wget -d” the plus-one upgrade again just as an imp would:

wget -d http://upgrades.electricimp.com/upgrades/0c2a69xxxxxx.2.dld
wget -d http://upgrades.electricimp.com/upgrades/0c2a69xxxxxy.2.dld
wget -d http://upgrades.electricimp.com/upgrades/0c2a69xxxxxy.2.dld

A transparent proxy that was too clever by half could conceivably return a redirect for that last fetch, knowing that it already had the content under a different name. Other than that, I can’t think what sort of thing would be going wrong here.

Peter

Hi Peter,

Performing your test - these are the 2 header-sets (in order) - and the payloads look fine.

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sun, 21 Sep 2014 20:30:18 GMT
Content-Type: application/octet-stream
Last-Modified: Fri, 05 Sep 2014 16:27:56 GMT
Accept-Ranges: bytes
Content-Length: 560880
Connection: Keep-Alive
Age: 0

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sun, 21 Sep 2014 20:30:23 GMT
Content-Type: application/octet-stream
Last-Modified: Fri, 05 Sep 2014 16:27:56 GMT
Accept-Ranges: bytes
Content-Length: 560880
Connection: Keep-Alive
Age: 0

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Sun, 21 Sep 2014 20:30:23 GMT
Content-Type: application/octet-stream
Last-Modified: Fri, 05 Sep 2014 16:27:56 GMT
Accept-Ranges: bytes
Content-Length: 560880
Connection: Keep-Alive
Age: 7

Hi,
I have the same problem too, 2 of my imp001 modules are stuck in firmware upgrade loop like the ones mentioned by others.
One other imp that I had used a long time back upgraded ok. Here are the Mac’s for the two that wont upgrade. I have tried all three of them on the same April board.
0c2a690024dc
0c2a690022b8

If i try http://upgrades.electricimp.com/upgrades/0c2a690024dc.2.dld from a webbrowser on my laptop connected to the same network as the Imp it downloaded a file 548Kb. (AT&T DSL in US).

Also tried connecting my imp to my wifi mobile hot spot didnt help, still stuck in firmware update loop.

Most strange that everyone is hitting upgrade problems with a vanilla http server all of a sudden! (especially as the upgrade got released weeks ago). Hmm.

Just as an experiment, I’ve touched the upgrade files on the servers so they all have last modified dates of just now. Wonder if that’ll help… give it a go?

Hugo thanks, no luck though, still stuck in firmware update.

Mine too, 12 quick red flashes, 3 slow green flashes, rinse and repeat.
This is from a known working home Internet connection that other IMP’s have used for upgades.

Device Logs - 234aa9eb6e4936ee
2014-09-22 18:44:09 UTC+10 [Status] firmware update triggered
2014-09-22 18:44:20 UTC+10 [Status] firmware update triggered
2014-09-22 18:44:31 UTC+10 [Status] firmware update triggered
2014-09-22 18:44:42 UTC+10 [Status] firmware update triggered
2014-09-22 18:44:53 UTC+10 [Status] firmware update triggered
2014-09-22 18:45:04 UTC+10 [Status] firmware update triggered

Then swapped in a different IMP into the same hardware on same wifi and:

2014-09-22 18:49:39 UTC+10 [Status] firmware update triggered
2014-09-22 18:51:33 UTC+10 [Exit Code] imp OS upgraded
2014-09-22 18:51:33 UTC+10 [Status] Downloading new code; 3.86% program storage used
2014-09-22 18:51:34 UTC+10 [Device] device firmware version: 9f88abe - release-30.15 - Fri Sep 5 09:11:38 2014
2014-09-22 18:51:34 UTC+10 [Device] starting

Hope this helps in the troubleshooting. Let me know if there’s anything else you’d like me to try.

Can you provide the mac address of the device having problems? The device ID doesn’t help very much if you’re swapping cards :slight_smile:

I believe we may have found a regression in the way we tell very old (2012) imps how to get an upgrade, which could account for this burst of issues since the last maintenance period. Looking into that now.

The MAC address of the IMP that was failing to complete the upgrade is 0c2a69012c7e. I’ll keep it unplugged for now, let me know if/when you’d like me to retry.

Hugo,
I just tried mine, MAC - 0c2a690022b6 and 0c2a690024dc
Still stuck in loop.
2014-09-22 19:00:07 UTC-4 [Status] firmware update triggered
2014-09-22 19:00:18 UTC-4 [Status] firmware update triggered
2014-09-22 19:00:28 UTC-4 [Status] firmware update triggered
2014-09-22 19:00:39 UTC-4 [Status] firmware update triggered

Good news, both of my imps got the firmware update fine now. Didnt make any changes to the network. Thanks for your help.

Yep, the fix server-side for old imps deployed, so this should be fine now. Apologies for the trouble, we’ll be adding some regression tests for this.

My upgrade-looped IMP is working again. An unusual message about booting to blank firmware but working and updated nonetheless. Cheers.

2014-09-23 18:59:55 UTC+10 [Status] firmware update triggered
2014-09-23 19:01:50 UTC+10 [Device] No firmware assigned; booting to blank firmware.
2014-09-23 19:04:19 UTC+10 [Device] device firmware version: 9f88abe - release-30.15 - Fri Sep 5 09:11:38 2014

That just means that you haven’t assigned any code to it yet.

My final Imp (236f7ceb6e4936ee) is still just flashing the “downloading” code…

2014-09-24 20:11:32 UTC+12 [Status] firmware update triggered
2014-09-24 20:12:27 UTC+12 [Status] Device disconnected
2014-09-24 20:14:28 UTC+12 [Status] firmware update triggered