Does anyone know of a good way to handle serial data ?

[SBSP]

[)roi(]
Regarding Option 1, Arduino's are slow they make use of a 16Mhz Crystal to clock an ATMEL Chip
If I had to format messages into JSON, I will actually make it a whole lot worse because I would be giving it a task to send more data via Serial. Maybe I'm missing your point but I don't see retrieving Arrays using JSON be any faster than using VB's String.Split function. the issue is not with the CAN-BUS Data, the company called SEEED Studio did a very good Job at retrieving CAN-BUS Messages , All I have to do is send it to the PC via Serial.


Option 2, Thats actually a great idea! never thought about that, I will put some thought into that, The way I see it the less information travel from the Arduino to the PC the better.

Option3 is what I'm actually doing in the example.

What GKM is saying, is the ultimate problem, and will be present in all above mentioned options.

A Layman example.
Machine A is sending Lucky Packet contents on a conveyor belt to Machine B in a factory.
1 Toy, 1 small packet of sweets and 1 lollipop makes a Lucky Packet and represents a CAN-BUS Message.
The conveyor belt represents the Serial Connection between th Arduino (Machine A) and the PC (Machine B)
Machine A Spits out, 1 Random toy, a pack of sweets and a random flavored Lollipop.
and it all travels to Machine B.

We have a worker in between Machine A and Machine B that Analyzes the the toys, sweets and Lollipops for defects running on the Conveyor Belt.

But the problem is the factory worker can only see 1 item at a time.
So if the worker does his job well he will always see Toy,Sweet,Lolipop,Toy,Sweet,Lolipop,Toy,Sweet,Lolipop.

But if you had to draw his attention onto something else, he will see, Sweet,Lollipop,Toy or Lolipop, Toy, Sweet when he focuses back on the convener, Simply because the Conveyer does not have a stop button.

Because my Previous Worker was a slow Piece of #%$# (HP H440p Laptop) it wasnt doing a good Job, But my newer worker Elite Desk 850 with a i7 4790K CPU is much better at doing the Job and lucky







By the way, currently I'm not making use of CAN-BUS data, I'm simply generating a stream of messages to emulate it. Otherwise i have to test it on my Jeep all the ime, which will be a huge pain.

Below is the Arduino Code.


The Emulator.

char incomingByte = 0;   // for incoming serial data
String BuildString = "";
int BuildingStatus = 0; //0=Not Building 1=Building 2=Done Building


String inBITS[9];

String PID;
String BIT1;
String BIT2;
String BIT3;
String BIT4;
String BIT5;
String BIT6;
String BIT7;
String BIT8;

String MyRND;

int RandomPID=0;



int led = 13;

// the setup routine runs once when you press reset:
void setup() 
{                
  
  Serial.begin(38400);    
}

// the loop routine runs over and over again forever:
void loop() 
{
  
 
  //PID="10438040";
  PID="1028";
  BIT1="100";
  BIT2="167";
  BIT3="10";
  BIT4="255";
  BIT5="255";
  BIT6="255";
  BIT7="255";
  BIT8="255";
  
  //Overwrite PID with a random val
  PID=random(1000, 1040); 
  MyRND = random(1,100);
  if (MyRND=="6")
  {
  BIT1=random(100,130);
  BIT2=random(100,130);
  BIT3=random(100,130);
  BIT4=random(100,130);
  BIT5=random(100,130);
  BIT6=random(100,130);
  BIT7=random(100,130);
  BIT8=random(100,130);
  }
  
  Serial.println(PID + "," + BIT1 + "," + BIT2 + "," + BIT3 + "," + BIT4 + "," + BIT5 + "," + BIT6 + + "," + BIT7 + + "," + BIT8 );
  delay(10);            
}

The Code to be used with the CAN-BUS shield.

//This Arduino UNO Sketch requires the Seeed CAN-BUS Shield Libraries
//https://github.com/yexiaobo-seeedstudio/CAN_BUS_Shield
#include <SPI.h>
#include "mcp_can.h"

INT32U canId = 0x000;

unsigned char len = 0;
unsigned char buf[8];
char str[20];


void setup()
{
    Serial.begin(38400);

START_INIT:

    if(CAN_OK == CAN.begin(CAN_125KBPS))
    {
        Serial.println("CAN BUS Shield init ok!");
    }
    else
    {
        Serial.println("CAN BUS Shield init fail");
        Serial.println("Init CAN BUS Shield again");
        delay(100);
        goto START_INIT;
    }
}


void loop()
{
    if(CAN_MSGAVAIL == CAN.checkReceive())  
    {
        CAN.readMsgBuf(&len, buf); 
        canId = CAN.getCanId();
        Serial.print("<");Serial.print(canId);Serial.print(",");
        for(int i = 0; i<len; i++)
        {
            Serial.print(buf[i]);Serial.print(",");
        }
        Serial.print(">");
        Serial.println();
    }
}

 

[[)roi(]]

Did you see the gist code? https://gist.github.com/AfricanSwift/b772917d38446880bf13a0aad2f3dd32
You're making your life unnecessarily difficult by converting unassigned char to CSV string and then back to a numerical value later. If you see my gist I suggest using Serial.write instead of print, this prints the equivalent character represented by the unassigned char value. Also you'll see I specifically avoid commas, it's not necessary.

 

[[)roi(]]

As for processing the serial stream; it's fairly simple to monitor for 9 characters, pull that off and process, alternatively you could add something like character 0x00 in between each group i.e. Keep reading stream until character 0x00, split and repeat... I'll modify the gist if you like to cover this?

 

[[)roi(]]

As for the JSON/HTML i.e. Option 1
What you've missed is that in this option all the code to summarise event IDs including status would be done on the Arduino, meaning the JSON served from the web server would only be the summary values, plus it could be enhanced for filters i.e. Only retrieve summary values for specified event IDs.

 

[SBSP]

Sorry I didn't see https://gist.github.com/AfricanSwift/b772917d38446880bf13a0aad2f3dd32
Let me have a look.

I'm not too familiar with source control, but I have uploaded a ZIP of the VB application here.
https://github.com/mailmartinviljoen/CAN-Opener

 

[[)roi(]]

Sorry I didn't see https://gist.github.com/AfricanSwift/b772917d38446880bf13a0aad2f3dd32
Let me have a look.

I'm not too familiar with source control, but I have uploaded a ZIP of the VB application here.
https://github.com/mailmartinviljoen/CAN-Opener

Cool I have a look a little later; a bit busy ATM
Btw it should be quite simple for you to convert your random value generating code; let me know if you need help with this -- I can post another gist

 

[SBSP]

As for the JSON/HTML i.e. Option 1
What you've missed is that in this option all the code to summarise event IDs including status would be done on the Arduino, meaning the JSON served from the web server would only be the summary values, plus it could be enhanced for filters i.e. Only retrieve summary values for specified event IDs.

I feel like a total idiot here, I cant see past the fact that I need to be able to seperate the message into 9 segments. How would the PC then know how to separate individual segments on the PC,
I.E
1000,111,222,333,444...

Do you mean I know that the ID is 4 digits long and the messages are 3 Digits/Characters long and there-fore I don't need the comma ?

 

[[)roi(]]

Don't worry I'll give you a better example later. Just got something to do now until 12.

 

[SBSP]

Don't worry I'll give you a better example later. Just got something to do now until 12.

Ok No worries, thanks for the help

 

[[)roi(]]

Ok let's start by defining what a group of data from the canbus will contain:

• canId : Unsigned 32 Bit Integer (UINT32)
• 8 values : Unsigned char

Let's see how this currently looks, using some values from one of your posts:

• <1536,2,26,136,0,0,0,0,0,>
• <1280,16,19,90,136,49,74,52,66,>

Serial transmission length:

• 1st line is a String of 26 characters
• 2nd line is a String of 32 characters

So to send both of these lines we would need to transmit 58 characters or 58 bytes of data; 1 character = 1 byte, (unassigned char)

Now let's look at the alternative I propose;
First point of note, all the values including canId are numbers, which means we can stored and transmit these as numbers as opposed to a String.

cnId	v1	v2	v3	v4	v5	v6	v7	v8
1536	2	26	136	0	0	0	0	0
1280	16	19	90	136	49	74	52	66

Ok now let's see how this will look when we transmit this data as an "unsigned char" i.e. as a number not as a String.

Number Bit Storage (1 byte = 8 bits; one or zero):
136 is stored as below with 1's and 0's. It is represented as the extended ASCII character ê

8	7	6	5	4	3	2	1
128	64	32	16	8	4	2	1
1	0	0	0	1	0	0	0

Note: Bit positions 8 and 4 have a 1, the rest have a 0. Total = 136
Ok the way this will be encoded is as follows:

• canId is UINT32, meaning it's storage space is 4 bytes or 4 unsigned char
• values 1 to 8 are unsigned char, meaning they each take 1 byte.

This means we need send a total of 12 bytes or 12 characters for this 1st line, and another 12 for the 2nd line (basically it will always be 12 characters, irrespective of the numbers or cnid).
Note: visually this not easily represented, as many characters are unprintable):

If we compare this against your CSV string version:

• 58 bytes versus 24 bytes.

However the benefits are more than just less resource; the conversion on the Arduino is simple and less taxing on the CPU; similarly the VB app will have to do less work to retrieve the values.

Group terminator:
In your original CSV String version; you utilised a comma to separate values and less than/greater than signs as a group terminator. Value terminators are not required in the alternative solution because the cnid will always be 4 bytes and the values will always be 1 byte i.e. 12 bytes per group. To identify the end of a group and to determine group value integrity I originally suggested using character 0x00 (i.e. NUL) as this is standard string terminator in the C / C++ world, however as the values can 0; we would have conflicts; a way around this could be to use 2 or 3 byte combination to signify a terminator. i.e. increase the transmit size from 12 to 15 bytes.

The Arduino support for Unicode appears limited, meaning Serial.write will still output ASCII characters i.e. instead of 1 unicode, so we would still end up with 2 ASCII characters; meaning no reduction on serial transmission size for packing. i.e. best option is what I've shown.

I have adjusted the code gist to simplify writing group(buffer write) and to reflect writing out the group terminator, which uses 3 x 255 i.e. last character in Extended ASCII set. https://gist.github.com/AfricanSwift/b772917d38446880bf13a0aad2f3dd32

Do you need some help on the VB end?

 

[SBSP]

Thanks , I admire your reply.
Just to test I'm trying to output some random data to the serial port with what you mentioned.

I declared

unsigned char terminator[3] = {255, 255, 255};

then simply I write terminator to the serial.

Serial.write(terminator);

But I get a compile error

exit status 1
call of overloaded 'write(unsigned char [3])' is ambiguous

If I write any value between 1 and 255 I get ASCII characters, If I use extended ASCII codes will it not be easier and just as quick to just convert them back to a number on the PC side ?

 

[[)roi(]]

Some pointers for the VB conversion;

• Basically you append the incoming data to a String variable checking if it contains the terminator; once found you split the String at the terminator into 2 variables left / right of terminator.
• Then check if the left portion contains 12 bytes (12 characters); if not discard this is incomplete / partial data. This should only ever happen at the start.
• The values right of the terminator is used as the starting point for the next incoming data from the serial port.

Making sense of the left 12 characters:

The 1st 4 characters represent the cnid;

• this conversion is a little more complicated re you need to treat 4 characters as representing a single value (the cnid); to convert this you will use Encoding.GetBytes to return a byte array for the 4 characters, and then use Bitconverter class to convert it to a UInt32 (https://msdn.microsoft.com/en-us/library/bb384066.aspx).
  
  example:
  

  ' The input String i.e. the 1st 4 characters in the 12 character group.
  Dim text As String = "VBTX"
  
  ' Convert String to Byte array.
  Dim array() As Byte = System.Text.Encoding.ASCII.GetBytes(text)
  
  ' convert the bytes array to a unsigned 32 bit integer
  Dim value As UInt32 = BitConverter.ToUInt32(array, 0);

The next 8 characters

• each represent a value between (0 to 255); to convert these characters to their numeric equivalent, you can use Asc() method to return the ASCII integer value the character represents.

Dim text As String = "x"
Dim value as Int = Asc(text)

Multithreading

• Fortunately the VB.Net SerialPort already pushes the data received event to a secondary thread i.e. so any code you add to this method will not block the main thread.
• Values you received from the buffer I think would be best stored in a string queue, each subsequent data receive event should be simply enqueued;
• Then get a separate thread to monitor the queue for new entries & append the values into a new String in the dequeueing order looking for the terminator sequence: 3 x 255 values, once you found a complete group split this off to string queue array in a round-robin sequence (1 queue per processing thread); these threads the undertake to convert the characters back to their numeric values and load it into your object graph.

Hope these pointers help. Let me know if you need a more detailed explanation, or some more code.

 

[[)roi(]]

for the "call of overloaded 'write(unsigned char [3])' is ambiguous" error try either of these:

unsigned char terminator[3] = {(byte)0xFF, (byte)0xFF, (byte)0xFF};

alternatively try this:

unsigned char terminator[3] = {(unsigned char)0xFF, (unsigned char)0xFF, (unsigned char)0xFF};

or this:

byte terminator[3] = {(byte)0xFF, (byte)0xFF, (byte)0xFF};

Note: 0xFF is hexadecimal for 255

 

[[)roi(]]

If I use extended ASCII codes will it not be easier and just as quick to just convert them back to a number on the PC side ?

The encoding is by default using extended ASCII considering that the return type of the canbus method

INT8U MCP_CAN::readMsgBuf(INT8U *len, INT8U buf[])

i.e. Unsigned Integer of 8 bits (values from 0 to 255)

Re the conversion on the PC side; see this post: http://mybroadband.co.za/vb/showthr...erial-data?p=17366812&viewfull=1#post17366812 for details of the steps to turn the character back into a integer value.

 

[animal531]

Pretty fast is debatable; in my experience C# is generally slower by a factor of 2 or more. It's never clear cut & a number of factors can throw this off, for example:

• Speed of the can-bus and/or including event frequency
• Event decoding and processing: higher workload = higher potential RAM / CPU usage
• Available RAM / CPU cores: less naturally equates to more GC pauses & higher potential for dropped events.

The difference as you probably know between C / C++ and .Net is that the former allows direct control over resource meaning performance can be guaranteed even on limited resource hardware e.g. an Arduino, secondly the overhead of .Net itself; the former can allow for offloading the process to the Arduino or the ultra tiny chip based computers, not so with .Net (which is beyond the typical capacity of these)

Ok, it sounded to me as if he'd be running the .NET app on his PC after passing the serial data there; but yeah, I wouldn't bother to run a sizable .NET app on anything less than a desktop.

However on desktop C# GC shouldn't ever be a bottleneck (at least until you get to an app which is allocating either most of the available free space, a ton of smaller allocations or is causing LOH issues).

 

[[)roi(]]

Ok, it sounded to me as if he'd be running the .NET app on his PC after passing the serial data there; but yeah, I wouldn't bother to run a sizable .NET app on anything less than a desktop.

However on desktop C# GC shouldn't ever be a bottleneck (at least until you get to an app which is allocating either most of the available free space, a ton of smaller allocations or is causing LOH issues).

Guess the practical side of me is a bit of a bastard for details.

 

[SBSP]

Thanks to everyone who helped so far, especially [)roi(] , I have a hectic week laying ahead, and will try what you said the coming weekend (if the wife allows me to )
thanks for the PM much appreciated!

 

[[)roi(]]

Thanks to everyone who helped so far, especially [)roi(] , I have a hectic week laying ahead, and will try what you said the coming weekend (if the wife allows me to )
thanks for the PM much appreciated!

Only a pleasure...