Are there any memory debug tools available in the imp OS? Of thousands of devices we’ve deployed, a few have started running out of memory and restarting lately. We haven’t pushed any code changes for months which makes me think it would be related to OS updates or just something really slow on our end.
I have devices running out of memory as quickly as daily but I’m unable to reproduce myself and it’s a very small percentage of the total fleet.
Other than just tracking the memory usage, are there any tools for actually monitoring things like abandoned objects, garbage collector actions, …? We have a lot of code and it’s going to be difficult or impossible to track down the bug without some sort of tool to help.
In previous cases we’ve helped debug, the issues have generally been related to hardware misbehavior (eg a pin toggling faster than expected due to a noisy input) which then causes code that behaves fine normally to start to leak memory - all fixable in squirrel, though.
GC happens automatically though you can force a pass. Objects with no references will be collected, so generally out of memory happens when there are references that you’re not aware of still being held unintentionally.
If you file a support ticket with some device IDs we can maybe do more debug? btw if you are seeing “memory error” (vs “out of memory”) that’s an impOS trap and we can look specifically at those issues as we get stack dumps etc.
It’s definitely “out of memory”. Is there any way to see a list of currently referenced objects? I’ll make a ticket with device IDs but they’re customer devices that I’m trying to get sent back so I won’t be able to try things on them for the moment. Do you guys have historic log access to something like this?
Hardware misbehavior makes a lot of sense to me too so that’s a good path to go down as well.