Plans for 1/4pin MCI or SPI driver and filesystem support?

In order to retain huge datasets, it’s often necessary to use storage medias such as SD cards.

Do you plan to add support for these?

Or do you plan to open up squirrel so that we can add libraries with extended functionality?

I have very good experiences with FatFS and implementing the SPI and MCI state-machines has been fairly easy on various Cortex M3 MCU’s I’ve used in the past.

The imp can do SPI natively: you could attach an SPI flash chip and talk to it from Squirrel. Though really the “imp way” is to keep any necessary “huge dataset” in the cloud somewhere, and have the imp itself be relatively dumb and just do whatever the cloud tells it to.

Peter

Hi Peter,

Thanks for your answer, but surely you don’t mean it would be smart to implement a FAT32 filesystem and the underlaying SD card state machine in Squirrel. The best choise for this, IMHO, would be as commands assessible from Squirrel but definitely implemented in C and perhaps in a few tight places in hand-optimized assemble.

When I’m talking about huge data sets stored on SD cards, I’m talking GB’s of data, which I don’t hope I have to first push up and then pull down from the cloud whenever I need to access them.

No, I wasn’t suggesting you implement SD in Squirrel! I was suggesting that if you had a few hundreds of Kbytes of data that for some reason needed to be on the card and not in the cloud, then you could use a small, cheap SPI NOR chip.

I don’t know what your exact application is, but if you have gigabytes of data, you can’t possibly be feeding the whole dataset in and out of the imp’s IO peripherals on any regular basis. So the dataset can surely live in the cloud, the “smarts” of your app can live in the cloud, and the imp can be a facility for “remoting the IO” from your cloud-based, gigabytes’ worth of app?

Peter

Thanks, Peter.
I can’t mention the exact application, only that (most of) the data is gathered, analysed and stored by my device, and it will later be retrieved and used by the same device. Only a small sub-set of the data will ever be needed to leave the device - this is where the cloud comes in.

Using an SD card for storage is the ideal solution for this kind of application and I’l really like to see this implemented on the imp.

Ah, okay. I am afraid that it’s starting to sound as if the imp is not really going to work for this application. First off, the imp’s CPU does indeed have an SD/SDIO peripheral – but it’s irrevocably committed to talking to the imp’s wifi chip, which acts as an SDIO slave. Secondly, there just aren’t the resources on an imp to manage a multi-gigabyte FAT32 filesystem. I looked at the FatFS web page, and it’s very impressive how little RAM it can get away with, but the performance if you can’t (say) fit an entire directory in RAM at once must be atrocious. (Not to mention the flash write amplification.)

Peter

I’m already using FatFS on a Cortex M3, and it’s working very well without ever fitting the whole directory in RAM. I think my max RAM usage is only about a few kiB - with double-buffered I/O and two files open - one read and one read and write at two different locations.
Even if the SD/SDIO peripheral is already in use, it’s still possible, as SD cards can also be accessed using regular SPI, and FatFS also works fine that way.

I really hope you will look into this, as it can be a very useful feature for lots of your users. - And it won’t be a huge effort for you to implement, if you use FatFS or similar filesystems.

(Don’t know why, but the discussion site kept popping up an “Error” “You don’t have permission to do that.” while I was writing this message.)

I’m afraid there are no plans to either implement FAT as an imp OS service or allow others to do this.

As Peter said, if you have gigs of data it really ought to be stored remotely. You could use an agent to talk to Amazon S3 and store the data there, for example. If this doesn’t fit with your application, then the imp likely isn’t suitable for it.

Thanks for your answer Hugo, although it was not the answer I hoped for.

Actually my application is already running perfectly on a Cortex M3 MPU, I’d just like to add wifi to it, and the imp seems to fit perfectly… I’d prefer only to have one MCU in the application, but surely I can degrade the imp to just be an advanced MAX232 and keep the rest of my current design. (It’s still a cheap way to add wifi to low-volume products)

And no, gigs of data should not necessarily be stored remotely. Only if it makes sense to do so - and in this case it doesn’t.

The comment about gigs of data should be stored remotely was referring to how best to build an imp-based setup; if you have a cloud server available, you should use it vs burden a device with this storage (assuming the application will work with that setup - as we don’t know your application, that could be the wrong call).

You also then get to back up the data, something you can’t do so easily with local storage.

Peter’s comment about flash write amplification does still hold, though. You’re likely generating a lot of unnecessary writes to flash due to the low memory design of FatFS. This may or may not be an issue with your application, but you should be aware of it.

As Peter said, if you have gigs of data it really ought to be stored remotely.

In a perfect world, yes. But suppose that the device is designed to wander in and out of the WIFI area… We would then LOVE to have some sort of mass storage option (SDCARD).

  1. pushing data is the way the IMP works today.
  2. a mix of store and then push (or partial push) is a legit method of cloud communication…
  3. store only should also be possible, why not?

We can read and write to an SDIO device (in a proprietary way) but FAT32 is a portable format and bodes well with removable micro/SD cards.

-Andrei

Yep, it sounds like you application is not really what we’re aiming at, which is why you’re running into limitations. As I said, no plans at all to add any local mass storage support, sorry.

Hugo, thanks for the note about write amplification. You are right, it’s a consequence of the small RAM footprint of FatFS, but it not really a huge concern in a small controller system.

A more valid concern to me, and to many others, is to have some sort of mass-storage available.

In my application I need to read/fetch and write/push at least 90kiB/s sustained. It’s easily done with FatFS and a SD card (currently MCI mode, but I believe SPI would perform equally well), but I’m not so sure about wifi and connection across the world will be able to do this without some sort of intermediate storage.

Sure, others don’t have the same (high?) requirements, but it still seems like you are downgrading and limiting your own system too much to really make it a stand alone system.

I hope you will seriously reconsider to add some sort of mass storage. Personally I’d prefer SDCARD, and I can certainly recommend to use FatFS (It’s a few days work, to integrate into FreeRTOS, and probably the same into whatever RTOS you have behind the blinds).