Time() accuracy on Imp005

General
1

Hi guys,

There have been a lot of posts over the accuracy of time() over the past few years and I just wished to check-in and see where it’s currently at with the v38 release on Imp005.

Assuming that the device has connected and had time to sync with the server and that there’s 32KHz crystal connected (if used), what’s the current accuracy and drift that can be expected on the Imp005 and are there any qwerks to be aware of?

Out of curiosity, does the Device talk directly to an NTP server or is there a bespoke mechanism used for syncing?

Thanks guys

2

The accuracy of any device when offline is based on the crystal used. Typically 32kHz parts, when tuned correctly (ie correct load caps) are ~20ppm, or about 2s per day.

When online, the time is synced with the server many times per hour, so as long as you’re processing inbound messages (ie not blocked for long periods) the imp will stay ~+/-1s or so from server time, which is NTP synced. Sync is over the imp TLS connection which accounts for why it’s not an NTP accuracy sync.

There were some bugs in earlier impOS versions (pre 34 possibly?) where time that drifted one way wouldn’t be corrected if the device was only online for very short periods, but those were fixed a while back. Currently there aren’t any known issues with time sync.

3

So this sort of got me thinking and I came up with something which is probably wrong in a hundred ways, but it’s an interesting experiment. See https://gist.github.com/hfiennes/07d5f8f53ed4e11d44b02071e9963c48

Essentially, this tries to provide a “more accurate” time on the device by first determining RTT latency (sending timestamps to the device which returns them). It collects 10 samples and checks the standard deviation - if this looks arbitrarily ok, it’ll use RTT/2 to offset the current server time, and sends it to the device. If it looks out of whack then it tried another 10 samples and so on.

The device, upon reception of one of these “corrected time” packets, notes the current hardware.millis() value, and then when get_time() is called, it’ll use the elapsed millisecond count to offset the last received time packet, and returns unix time and microseconds to the application.

For example purposes, the device prints out pre-sync time and post-sync time every time a “corrected time” packet is received, so you can see how much correction is being applied each cycle (the first output is bogus to keep the code clean).

Note that as wifi is hugely variable, this jumps around a lot… an example from an imp004m on my desk:

|2019-01-24T17:57:33 -08:00|[Device]|pre 1548381453.602633 -> post 1548381453.605479, diff 2846us|
|2019-01-24T17:57:44 -08:00|[Device]|pre 1548381464.009479 -> post 1548381464.009072, diff -407us|
|2019-01-24T17:57:50 -08:00|[Device]|pre 1548381470.069072 -> post 1548381470.069140, diff 68us|
|2019-01-24T17:57:59 -08:00|[Device]|pre 1548381479.676140 -> post 1548381479.670571, diff -5569us|
|2019-01-24T17:58:09 -08:00|[Device]|pre 1548381489.665571 -> post 1548381489.671222, diff 5651us|
|2019-01-24T17:58:20 -08:00|[Device]|pre 1548381500.726222 -> post 1548381500.722919, diff -3303us|
|2019-01-24T17:58:33 -08:00|[Device]|pre 1548381513.003919 -> post 1548381513.008424, diff 4505us|
|2019-01-24T17:58:40 -08:00|[Device]|pre 1548381520.041424 -> post 1548381520.041819, diff 395us|
|2019-01-24T17:58:50 -08:00|[Device]|pre 1548381530.123819 -> post 1548381530.120787, diff -3032us|
|2019-01-24T17:59:00 -08:00|[Device]|pre 1548381540.015787 -> post 1548381540.017660, diff 1873us|
|2019-01-24T17:59:10 -08:00|[Device]|pre 1548381550.057660 -> post 1548381550.059011, diff 1351us|
|2019-01-24T17:59:20 -08:00|[Device]|pre 1548381560.080011 -> post 1548381560.079810, diff -201us|

And here’s one from an ethernet-connected imp005:

|2019-01-24T18:06:55 -08:00|[Device]|pre 1548382015.776917 -> post 1548382015.776352, diff -565us|
|2019-01-24T18:07:05 -08:00|[Device]|pre 1548382025.772352 -> post 1548382025.770901, diff -1451us|
|2019-01-24T18:07:15 -08:00|[Device]|pre 1548382035.772901 -> post 1548382035.771540, diff -1361us|
|2019-01-24T18:07:25 -08:00|[Device]|pre 1548382045.776540 -> post 1548382045.775398, diff -1142us|
|2019-01-24T18:07:35 -08:00|[Device]|pre 1548382055.768398 -> post 1548382055.767375, diff -1023us|
|2019-01-24T18:07:45 -08:00|[Device]|pre 1548382065.770375 -> post 1548382065.769359, diff -1016us|
|2019-01-24T18:07:55 -08:00|[Device]|pre 1548382075.772359 -> post 1548382075.771143, diff -1216us|
|2019-01-24T18:08:05 -08:00|[Device]|pre 1548382085.779143 -> post 1548382085.778079, diff -1064us|
|2019-01-24T18:08:15 -08:00|[Device]|pre 1548382095.766079 -> post 1548382095.765539, diff -540us|
|2019-01-24T18:08:25 -08:00|[Device]|pre 1548382105.774539 -> post 1548382105.773151, diff -1388us|
|2019-01-24T18:08:35 -08:00|[Device]|pre 1548382115.766151 -> post 1548382115.764830, diff -1321us|
|2019-01-24T18:08:45 -08:00|[Device]|pre 1548382125.771830 -> post 1548382125.771531, diff -299us|

The offsets here are all lower, but also all in one direction which implies to me that the 005 hardware.millis() is running fast… curious (and something we’ll investigate).

Real NTP does a lot of stats on the time corrections to remove outliers and slowly slews the clock to what hopefully is the right time. None of that is happening here - if the RTTs look stable but the correction packet takes forever to arrive, the device just believes it and the time will be wrong. With more time and ability to understand stats then someone could do a better job of removing outliers on the device side and slewing too.

Still, something fun to play with.

4

Thanks for the response Hugo and the detailed look into the matter, we’ll continue to explore this approach ourselves too as timing is important to us.

We also noted our Imp005s counting faster, this post was prompted by an issue where we would observe the Imp005 slowly creep ahead of the server time, receive a sudden correction and jump by a second every few hours so it would be good to hear back about your findings on this.

Thanks again

5

Time, it turns out, is a complex problem.

Time-of-day in agent Squirrel is set by NTP in our cloud-servers, so it is monotonic, accurate, and free of systematic drift.

Time-of-day in device Squirrel where the device is mostly online, is set from the server with a one-second precision as Hugo describes. It is not monotonic; can be between one second fast and N seconds slow, where N is the “ping time” to our servers (in whole seconds, rounded up); and drifts with the 32kHz crystal frequency as Hugo describes (20ppm), except on imp005 where it drifts with the frequency of the (internal) 37.4MHz crystal.

Time-of-day in device Squirrel on imp001-004 when the device is mostly offline but has a 32kHz crystal, is set from the server: it is not monotonic, and can be up to one second fast, or up to 120 seconds slow. (If the imp sees a “too new” time then it knows it must catch up; if it sees a “too old” time then it can’t distinguish clock drift from packet delay.) It drifts with the 32kHz crystal. (Or at least, that was the original design; a subsequent server-side change deliberately “defeated” this mechanism by sending several time events back-to-back, meaning that nowadays such imps are likely to run a variable amount slow depending on instantaneous network latency, but will never get to 120s slow. That change improved accuracy in good network conditions, but can also reduce accuracy and monotonicity in bad or inconsistent network conditions. Note that even NTP would struggle to maintain accuracy if only occasionally and briefly connected to a network.)

Time-of-day in device Squirrel on imp005, or where the device does not have a 32kHz crystal, is completely incorrect unless/until the imp connects to the server each boot, and then follows the previous scheme, except that the drift is now that of the “fast” (CPU clock) crystal, and these tend to be more drifty than 32kHz crystals – 30ppm or more. (Though, having said that, the imp005 one appears to be specified as 10ppm.)

The microsecond counter returned by hardware.micros (and indirectly by hardware.millis) is monotonic (except when the imp restarts, and when 32-bit integers wrap), and drifts with the fast crystal. No attempt is made to synchronise time-of-day and the microsecond counter on devices with two crystals, so they drift independently because the two crystals drift independently.

Better time-of-day clocks (MSF, GPS etc) can be straightforwardly added to the imp as external peripherals, but are not cheap.

Peter