We are quickly hitting 100% of program storage and are wondering how others have addressed that problem in the past. Any best practices to share?
I already learned directly from EI that (as expected) comments and empty lines don’t count towards the program storage, but variable and function names do (although assumably not too much, due to a lookup table).
What other tricks are you guys using?
Thanks!
Also, what imp are you on? 001/002 have half the space of the other devices (128kB vs 256kB)
It’s an imp005, so it is really curious that we managed to fill it all up. More than 25% comes from BGLib and related code, I know that already.
Strings declared in the device code can eat up a lot of programme memory. Since the imp005 has lots of RAM, could you consider storing strings in flash and loading them as resources into RAM at startup?
Also, looking at BGLib (which I’ve never used), it seems to be written for clarity, not brevity. I guess you would expect that for a publicly available library.
After reviewing BGLib, I can see a very useful technique that I’ve not seen documented by Electric Imp, or in the original Squirrel docs.
I have been constantly frustrated by the inability to declare consts and enums within classes or tables. The bytecode compiler has never permitted it. This forces two options: 1. declare them outside of the class/table (messy and less portable), 2. declare constants as static slots within the class. (eats program memory, but have seen this technique in many Electric Imp library and example classes).
BGLib declares consts and enums within a class method. The compiler has no problem with that. It seems an oversight, rather than a feature. Those consts and enums are not limited in scope to the class, they can be referenced externally (post declaration). They take up no memory, if not used. They allow all relevant stuff to be rolled up into a single class. Lastly, the function that they sit in never actually needs to be called (although it is in BGLib).
I think this quirk should be included in the documentation about efficient squirrel.
You can declare consts and enums lexically inside a class, yes. But as you say, they aren’t scoped to the class – the effect is exactly the same as declaring them outside the class.
This is not a “useful technique” – it’s a misleading way to use an existing technique. Note that the writer of BGLib has, very responsibly, prefixed all the enum and class names with “BLE_” so as not to conflict with other libraries.
Peter
In the context of this thread (saving programme memory), I think it is useful. With a few exceptions, like BGLib, Electric Imp sets the scene with its libraries using static strings declared within classes, rather than declaring them outside the class as consts. Static strings reserve memory space for the strings and their keys. This practice adds up if you are implementing lots of classes. Those strings are allocated regardless of whether they are actually used, whereas the the consts and enums aren’t. I believe that some of this was done this way because it is desirable, and responsible, to author self-contained classes. This is particularly the case with Squirrel, because we don’t have the luxury of being able to divide up our code into separate files in the IDE. Curating 10K+ lines of Squirrel is a pain without it.
Embedding the consts and enums inside a function feels bizarre, but it serves a purpose. I’m not saying that every situation can be solved with a constant.
In a more general context, Electric Imp’s userbase includes those who may have come from a web background and those who’ve come from an embedded background (me). Electric Imp has done very well to implement a VM with such a small footprint. However, we are all still dealing with a device with limited resources. The illusion of including lots of libraries and using elegant syntax cracks when users hit the programme space limit. Trade-offs are necessary.
@thomaskoehn, sorry to hijack your thread!
@coverdriven 
no worries. happens all the time 
I just commented out all our logging. We do quite extensive logging with a custom solution. Commenting out these roughly 500 lines contributed to a decrease in program storage by 12%. Sounds unexpectedly much for me. I guess it is really helpful to keep strings as short as possible. Or what is the conclusion from your side, @peter @smittytone?
Sorry yes, I’m not suggesting replacing those enums and consts with static strings. You’re absolutely right that doing so would dramatically increase memory usage. I’m just suggesting that it’s more “honest” to move those enums and consts out of the class definition, because people seeing them inside the class definition might wrongly conclude that, like everything else inside a class definition, the enums and consts have class scope.
On the logging/BGLib question: all those strings in BGLib (event.name and the first parameter to command.send) are only there for logging purposes; they don’t form either part of the API or part of the BTLE protocol. In each case they can be reconstructed from the rest of the command/event, and add no new information. If it was me, I’d take all of them out, and if I wanted more verbose logging I’d write a post-processor that took in textual log data and turned the msg_type and cid fields into more readable names. There’s no reason to have that name lookup taking space on the device.
Peter
@thomaskoehn, this might sound like a small detail, but it can also be helpful:
On top of keeping your strings smaller, use consistent key names across classes and tables. This will result in less memory use. For instance, if you have classes AClass and BClass, try to give them the same method and property names where it makes sense to do so, such as “init”, “add”, “key”, “config”, “verify” etc. There is no risk of name clashes because of their limited scope. I believe that the squirrel bytecode compiler will reuse those strings rather than having to allocate something distinct.
I don’t think this matters with local variables unless they span closures, but I could be wrong about that.
@peter I did a few experiments around the informative strings in BGLib (all percentages are for imp005):
event.name <- lines saves about 1%All in all, these results are not promising at all. My current game plan is to remove all functionality from libraries that we don’t use.
@peter @hugo
There is one item, though, were I expect Electric Imp to provide support in the near future:
Our custom AES-128 implementation uses a full 14% of program storage, because kind of matrices are statically put into the library. This is done on purpose because I prefer to statically run out of program storage and know at time of upload, then to run into out-of-memory issues at runtime.
When can be expect AES-128 to be natively supported on the imp?
This is not a technical issue, it’s more an export controls issue I’m afraid. Yes, that makes no sense but we have to be careful to ensure that we can keep the 5A992/5D992 categorization for the imp hardware.
How close are you to 100% on local code?
I’m developing a bigger update right now, which is due in the next couple of weeks, and in the development process I crossed the 100% already a couple of times. To mitigate this, I removed methods that I would actually need in production. So it is really tight 
Using imp005 or another one?
imp005, yes.
(adding some more characters to meet the minimum length of a post…)
How are your AES tables represented? Strings and blobs are much more compact than Squirrel arrays.
Peter
oh, maybe you found something there: it is several large arrays of blobs. What would you recommend? Giant strings instead? Actually, you can just have a look on your own. We made it public a while ago:
I reckon that if you start replacing those arrays with blobs initialised by giant strings then you’ll see a big reduction in code size. If you can bear to replace them completely with giant strings (not blobs), you’ll see a reduction in runtime RAM usage too.
Peter