Serial from Arduino to Imp loses characters

Squirrel Code
1

I am sending about 100 characters of data from my Arduino to an Imp Shield (has level shifters) using Software Serial and Serial.print commands. The Imp only seems to be able to read the first 79 characters and then loses the rest. Here is some of the code from both

Arduino Code:

#include <SoftwareSerial.h>
SoftwareSerial impSerial(8, 9); // RX on 8, TX on 9

void setup() {
impSerial.begin(9600);
}
void loop() {
// Output to Imp

impSerial.print(“W,”); //Start Character: There is never a W character in the data.
impSerial.print(Data1);
impSerial.print(",");
impSerial.print(Data2, 1);
…
impSerial.print(“X”); //Stop character: There is never an X character in the data.

Imp Device Code:

hardware.uart57.configure(9600, 8, PARITY_NONE, 1, NO_CTSRTS);

function pollUart()
{
//Tried lots of values for the polling rate. This is really slow. Don’t care about blocking other code
imp.wakeup(2.00, pollUart.bindenv(this));
local bcnt = 0;
local s = “”;
local byte = hardware.uart57.read(); // read the UART buffer
// This will return -1 if there is no data to be read.
while (byte != -1) // otherwise, we keep reading until there is no data to be read.
{
s+=byte.tochar(); // Add byte to string s
bcnt++; // add 1 to byte count
byte = hardware.uart57.read(); // read from the UART buffer again (not sure if it’s a valid character yet)
}

if (bcnt > 0){
local len = s.len();
server.log(s);  //Print out the string for debugging - shows lost characters when longer than 79 characters
if (s[0].tochar() == "W" &&  s[len-1].tochar() == "X") {

server.log("String received Good W and X");

local sp = [“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”];
local pr = [“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”];
sp = split(s,",");
}
pollUart(); // start the UART polling, this function continues to call itself

2

You are polling the uart too slowly. The 80 character input buffer is filling up in the 2s you are waiting between polls.

ie, your handler enters, reads a whole buffer, runs out, then sleeps for 2 seconds in which time another 2*9600/10 = 1,920 bytes could arrive.

You should really be using the serial callback, or at the very least reduce your callback time significantly. You’d also need to alter your algorithm to not expect to receive an entire transmission in one go - right now, every time you enter pollUart you expect to receive the while transmission (ie you start with a blank string). You should have a global string and just add to it in pollUart, and process it if you receive an “X”.

3

Try this (written blind, but…)

`// string we’re collecting
s <- “”;

function process(newline) {
server.log(newline);
}

function newdata() {
local b = hardware.uart57.read();
while(b!=-1) {
if (s == “”) {
// We look for an S to start the string, otherwise ignore
if (b == ‘S’) s = “S”;
} else {
// Append to string
s+=b.tochar();

  // If we saw an X, we got the end of the string
  if (b == 'X') {
    // process string
    process(s);
    // and blank it
    s = "";
  }
}
b = hardware.uart57.read();

}
}

// Init uart. It will call callback
hardware.uart57.configure(9600, 8, PARITY_NONE, 1, NO_CTSRTS, newdata);
`

(edited 20140428 as I was missing a close brace and the re-read of the UART was at the wrong nesting level… but not bad for being written blind!)

4

You make some excellent points and expose my newbieness.

I copied a lot of this code from another script that had a MUCH shorter callback time of .001 or even .0001, but then increased it when I started losing data. Clearly, I was trying to mask a more fundamental problem.

By using serial callback instead of polling, do you mean I should use something like the following and then have my “arduinoData” function have a better algorithm where I don’t expect the entire transmission in one go?
hardware.uart57.configure(9600, 8, PARITY_NONE, 1, NO_CTSRTS, arduinoData);
I recently read this article that should help me write that better algorithm:
http://www.gammon.com.au/forum/?id=11425
He says “Something that seems to “throw” beginners to programming on a microprocessor is how to receive incoming serial data without “blocking”. That is, being able to do other things while the text is arriving.” and that you need to add buffering. Here is his sample code:
`
/*
Example of processing incoming serial data without blocking.

Author: Nick Gammon
Date: 13 November 2011.
Modified: 31 August 2013.

Released for public use.
*/

// how much serial data we expect before a newline
const unsigned int MAX_INPUT = 50;

void setup ()
{
Serial.begin (115200);
} // end of setup

// here to process incoming serial data after a terminator received
void process_data (const char * data)
{
// for now just display it
// (but you could compare it to some value, convert to an integer, etc.)
Serial.println (data);
} // end of process_data

void processIncomingByte (const byte inByte)
{
static char input_line [MAX_INPUT];
static unsigned int input_pos = 0;

switch (inByte)
{

case '\

': // end of text
input_line [input_pos] = 0; // terminating null byte

  // terminator reached! process input_line here ...
  process_data (input_line);
  
  // reset buffer for next time
  input_pos = 0;  
  break;

case '\\r':   // discard carriage return
  break;

default:
  // keep adding if not full ... allow for terminating null byte
  if (input_pos < (MAX_INPUT - 1))
    input_line [input_pos++] = inByte;
  break;

}  // end of switch

} // end of processIncomingByte

void loop()
{
// if serial data available, process it
if (Serial.available () > 0)
processIncomingByte (Serial.read ());

// do other stuff here like testing digital input (button presses) …

} // end of loop
`

5

@tweil >>By using serial callback instead of polling, do you mean I should use something like the following and then have my “arduinoData” function have a better algorithm where I don’t expect the entire transmission in one go?

Yes that’s it and looks like Hugo’s done that for you in his example…

Also have a read of Tony’s excellent event programing
https://electricimp.com/docs/resources/eventprogramming/

6

Hugo’s code works like a charm and is simpler that Gammon’s code I posted. This method makes a lot more sense than what I was trying to do. I’ve now been able to increase the amount of data with no problem.

The one very minor thing I don’t understand is that my data starts with a “W” and when the code sees this it assigns the “W” to the beginning of the blank string. When I print out the string there is “WW,Data1” at the beginning rather than the “W,Data” I would expect. It’s not a problem since I discard that anyway, but I’m curious to understand it.

7

Oh, I thought it started with an S. Change the ‘S’ to ‘W’ in my code (I suspect you have already otherwise you’d not see it working at all!). Not sure why it’d double the first byte though. Sure the sender isn’t sending two W’s?

8

I’m absolutely positive I’m just sending one “W” from the Arduino. I changed the code on the Imp to add an “S” instead of “W” and now I’m getting an “SW,Data”. I throw away that first piece of data anyway, so it doesn’t really matter, but I really want to understand out why it is doubling up the first byte.

9

Definitely copied my code exactly? Can you copy/paste yours? Missing the else would double up.

10

Thanks for taking a look at it:

`
function arduinoData() {
local b = hardware.uart57.read();
while(b!=-1) {
if (s == “”) {
// We look for an S to start the string, otherwise ignore
if (b == ‘W’) s = “S”;
}
else {
// Append to string
s+=b.tochar();

  // If we saw an X, we got the end of the string
  if (b == 'X') {
    // process string
    process(s);
    // and blank it
    s = "";
  }
b = hardware.uart57.read();

}
}
}
`

data looks like this:

2014-04-28 22:21:28 UTC-4: [Device] SW,72.9,50.7,29.77 Falling,72.3/74.8,42.8/54.4,30.04/30.00/29.93/29.91/29.93/29.83/29.85/29.78,X

11

Ah, I typed it wrong. Move the b=hardware.uart57.read(); down one set of braces. This meant that it wouldn’t pull the next character from the UART when it saw a valid string start and explains the doubling.

I’ll edit my code above to be correct.

12

That did it. I’m embarrassed that I couldn’t find that. I appreciate that you did. Works like a charm now!

13

I have a question about the arduino sketch; I don’t see where you actually declare either Data1 or Data2. I realize you’re probably reading from some sensor(s), but can you paraphrase your variable declarations here so that we can see the basics?

Thanks!