Does anyone know of a good way to handle serial data ?

[SBSP]

I have a micro controller that sends data via serial. Regardless of the programming language that will be used to receive data. What's the best way to handle incoming serial data that does not care if you received the data or not.

This will be my 3rd attempt, the first attempt was using VB6 then I switched over to VB.NET and will again be using VB.NET.

This is how it works, I connect a CAN-BUS Transceiver to an Arduino UNO, The transceiver basically eaves drop on flyby data.

I.E
In modern cars you have a CAN-BUS and Device A (Engine ECU) sends data
Speed,
RPM,
Safety belt plugged in or not,
Tyre Pressure,
Distance traveled,
Lights on or off,
Current Gear Selection and a whole bunch more

Device B (Doors which has central locking)
Device B receives and ignore all the data being sent by the ECU because none of the messages are intended for it.

Device C (Instrument cluster)
Receives all data sent by Device A and because the data is meant for it it takes the information and displays it accordingly on the instrument cluster.

A typical CAN-BUS message looks like this.
000,000,000,000,000,000,000,000,000
each bit has a value between 0 and 255 except for the first bit which can range from 000 to 9999
The first bit is the ID of the message.

i.E

Device A = 1034
Device B = 1021
Device C = 1044

The rest of the data is structured in various ways, but for example purposes Device C is structured like this.

1044,10,0 = Lights OFF
1044,10,1 = Lights ON
1044,11,0 = Safety Belt Not plugged in
1044,11,1 = Safety Belt plugged in
104412,0 = RPM meter position 0
104412,1 = RPM meter position 1
104412,2 = RPM meter position 2
104412,3 = RPM meter position 3
104412,4 = RPM meter position 4 ...... ect ect

The biggest problem is to ensure I don't miss information.
when I monitor the bus with putty it will create a megabyte of text data in no time.

This is what I usually do, I output the serial data using the Arduino in CSV format with a leading < and trailing > like this.
<000,000,000,000,000,000,000,000,000> then send it over serial to my PC.

And I then receive it into a buffer string and by searching for "<" and ">" and at the same time keeping track of the cursor I'm able to track all messages that flew by, Sometimes the buffer fills up and the application crashes, So i discard all data before the cursor.

The problem with the above is its based on the clock speed of the PC CPU, and after adding a whole bunch of features like building a list of ignored IDs and ignoring complete but unwanted messages , updating a grid with incoming messages by first finding it in the grid then update or if it isnt in the list ad a new message, doing all that slows things downwhich also means that I sometimes have to sacrifice messages that flew by for CPU time.

How would you have done the above ?

 

[Beachless]

I must say I dont think I fully understand what you are doing and Im typing on my phone but I will give it a go...

First of all trying to read all that info into a string(if thats what you are doing) is a really bad idea. Secondly you dont talk about any threading? Computers are not great at realtime processing thats where microcontrollers thrive but you can improve things by creating a thread dedicated to reading the serial data and deciding what to do with it.
I would suggest firstly using the arduino to dump useless/too much info or worse case the dedicated thread. Things like temperature can be read every second for instance and then dump the rest of the temperature mesages until your timer says you need to read them again. Or the arduino can read the temp and compare it to the last temp and only send the updated message if there was a change.

In short use the arduino to limit serial data passed to to pc as best you can. Next use a dedicated thread on the pc to read the serial port and further filter data until you have only what you need.
Send this data to another thread to process in whatever manner you need.

Maybe let us know what your end goal for the data is that may assist us a bit more. Sounds like a fun project.

 

[SBSP]

Before I explain what the goal is, I need to add the following.
CAN-BUS networks are in a way the same as a normal Ethernet star topology. It sends data to all devices only the devices on the network will only make use of information if it was intended for them, by reading the ID. The Arduino equipped with a CAN-BUS shield can be seen as WIN-PCAP driver who behaves like Etherreal who will monitor all data regardless if it was intended for it or not.

So to get back to your comments, I want to make a tool that will take all information that flew by and dump it into a grid.
But the information to be dumped into the grid needs to group by the ID, so that you dont have to look at data matrix style.

Unfortunately it has to be realtime, because the incoming data is to be used to reverse engineer messages, I dont know if a message is for temperature or if its for Speed or RPM ect ect.

I made the previous tool freely downloadable from here
See it here http://www.techtinker.co.za/forum/viewtopic.php?f=14&t=18 (You have to register to see the pictures in the forum)

The reason why it has to be realtime is, I physically watch the grid, when things are forever changing without pressing any butons in the car I know its not data I'm interested in, SO I would run the console for about 1 min, then right click and ignore the message ID I think is junk like the current time (seconds causes it to change all the time), Once I got rid of ID that change all the time, I start pressing buttons and as I press them the ID changes as well, that's how I know I'm fiddling with the correct ID.

From there I save the messages and replay them back into my Jeep which then allows me open doors, enable an disable diff lockers.
My next challenge is to find the ID that turns on the Flat tyre icon. I upgraded my tyres from standard to 34" 305/70/17" wheels which now doesnt have the tyre pressure sensors anymore. So It thinks the tyres are flat, There are a few other things I want to do like stop the engine fan when crossing deep water ect ect. besides all of the practical stuff, I like messing with the gimmics.

Here is an example where I'm showing how to integrate aftermarket accessories.
In the below video I'm able to click a relay when I put the highbeams on, the arduino and CAN-BUS shield is hooked up to the BUS wires on my Jeep's Radio.

[video=youtube;YK2GefWiG4w]https://www.youtube.com/watch?v=YK2GefWiG4w[/video]

 

[Magnum]

How does the Tyre sensors work? Why hack the whole system when you only need to hack a single sensor? or 4 sensors of the same kind?

You are making this extremely difficult for yourself. take a sensor get the output ranges, build a signal generator that outputs the signal for tyre inflated and input that into the sensor signal wire. Problem solved.

Serial data is a mess the decoder must match the encoder with the right inputs and outputs in the right order. then there are also parity checks on the serial signal. this is a can of worms not even I will attempt.

hence me designing a control circuit for my cobra from the ground up.

 

[SBSP]

How does the Tyre sensors work? Why hack the whole system when you only need to hack a single sensor? or 4 sensors of the same kind?

You are making this extremely difficult for yourself. take a sensor get the output ranges, build a signal generator that outputs the signal for tyre inflated and input that into the sensor signal wire. Problem solved.

I don't think you get the Idea, I'm not only after the sensors as i said it just the next challenge, besides that I don't have the sensors anymore., Suppose If I still had them could pressurize them in a PVC pipe and dump them into the boot.

Its also not entirely solving a "Real" problem, It simply having fun messing learning how it all works in "Just because I can kind of way",

 

[Messugga]

I think you're trying to reinvent the wheel here. CAN-BUS hacking isn't new and as I'm sure you know, implementations differ between models of vehicles and whatnot.
What this does mean though, is that other people have built some tools to help you reverse engineer some things.

When I played with it a while back on my 86, this site gave me some useful links:
http://chadgibbons.com/2013/12/29/hacking-the-jeep-interior-can-bus/

 

[Magnum]

Its not off topic. It's realizing there are more than one way to tackle a problem and find a solution for it. play is fun. you learn a-lot.

And you missed my edit. all things can be done. with enough time and devotion. There is no greater feat then acquiring knowledge.

with fiddling you can send a incorrect signal which the can system can lock up all functions and need a factory reset or maybe even replacement. be careful.

 

[SBSP]

with fiddling you can send a incorrect signal which the can system can lock up all functions and need a factory reset or maybe even replacement. be careful.

I know I have lost quite a bit already by messing around, I ran the battery down too many times and had to replace it, I also broke a Programmer which can remove things like TPMS icon. I hacked it to pieces
I drilled into the casing solders wires onto it ect ect.




and also somehow locked it to a fictitious VIN and it cost around R3500 to replace. Broke a couple of ELM327 scanners.

The first time I sent programmable data into the OBD2 port was the last time.
It did nothing except hooting for about 30 seconds without being able to stop it.

One of the below will disable TPMS (Or rather set the threshold very low) I recorded it from the procal. When using the procal as you are supposed to it will program then honk twice, Somewhere it must have missed something so instead it honked for around 30secs
But I wont be trying that again.

Enter setting mode success
set rate success!!
Enter Normal Mode Success!!
CAN BUS Shield init ok!
<1536,2,26,136,0,0,0,0,0,>
<1280,16,19,90,136,49,74,52,66,>
<1536,48,0,0,0,0,0,0,0,>
<1280,33,69,54,68,49,52,65,76,>
<1280,34,50,48,54,48,52,52,0,>
<1568,2,33,10,0,0,0,0,0,>
<1284,16,18,97,10,240,2,17,2,>
<1568,48,0,0,0,0,0,0,0,>
<1284,33,10,4,0,3,0,0,34,>
<1284,34,0,0,0,2,3,0,0,>
<1568,2,33,176,0,0,0,0,0,>
<1284,16,20,97,176,2,0,0,0,>
<1568,48,0,0,0,0,0,0,0,>
<1284,33,0,0,0,0,68,194,234,>
<1284,34,0,225,0,255,255,15,0,>
<1568,2,33,178,0,0,0,0,0,>
<1568,48,0,0,0,0,0,0,0,>
<1284,33,0,0,0,0,0,194,234,>
<1568,2,33,18,0,0,0,0,0,>
<2015,2,62,2,0,70,84,87,33,>
<1568,2,16,146,0,0,0,0,0,>
<1284,2,80,146,250,1,192,4,215,>
<1568,2,33,11,0,0,0,0,0,>
<1568,48,0,0,0,0,0,0,0,>
<1284,33,1,2,3,0,255,156,128,>
<1284,34,30,147,97,11,14,0,133,>
<2015,2,62,2,0,70,84,87,33,>
<1284,16,11,97,178,0,0,0,0,>
<1568,2,16,146,0,0,0,0,0,>
<1284,2,80,146,97,11,14,0,133,>
<1696,2,16,146,0,0,0,0,0,>
<1300,2,80,146,7,0,0,0,0,>
<1568,4,48,173,7,1,0,0,0,>
<1284,3,112,173,7,11,14,0,133,>
<1696,4,48,64,7,1,0,0,0,>
<1300,3,112,64,7,0,0,0,0,>
<1568,4,48,173,7,0,0,0,0,>
<1284,3,112,173,7,11,14,0,133,>
<1696,4,48,64,7,0,0,0,0,>
<1300,3,112,64,7,0,0,0,0,>
<1568,4,48,173,7,1,0,0,0,>
<1284,3,112,173,7,11,14,0,133,>
<1696,4,48,64,7,1,0,0,0,>
<1300,3,112,64,7,0,0,0,0,>
<1568,4,48,173,7,0,0,0,0,>
<1284,3,112,173,7,11,14,0,133,>
<1696,4,48,64,7,0,0,0,0,>
<1300,3,112,64,7,0,0,0,0,>
<1924,2,17,130,0,0,0,0,0,>
<1536,2,17,130,0,0,0,0,0,>
<1280,3,127,17,18,48,52,52,0,>
<1925,3,127,17,128,0,0,0,0,>

But sending static non programmable messages is pretty safe. Regarding the blog post you sent , I contacted Chad Gibbons long ago when he first started the blog post and he helped me quite a bit, I corrupted the BUS by using the wrong baud rate.

So yes I know of all the available tools I.E the AEV Procal . all of them cost money and the software is free as long as you buy their expensive hardware. Different models do have different message sets but are usually grouped by a couple of years, Chad's Jeep has totally different messages to mine he has a pentastar and I have the 3.8L

All of this can also lead to better integration between donor engines most people who stuff a lexus V8 into a car never get the instrument cluster to work properly. My Idea is to make a free tool available for everyone to use and modify as they like. The Can-Bus Sniffer V1.XX which I have created before has already helped quite a bit of people and driving me nuts to get it to work due to VB6 run times not being installed when they use it. the Serial communication issues are present in it but still very usable and successful at capturing data. I just want it to work better in VB.NET which allows for easier installation on a PC.

 

[[)roi(]]

The solution appears fairly obvious.

You need to read up on multithreading. For example:

• 1 thread would focus solely on receiving the data and storing this in memory (array buffer), queue structures seem to be an obvious choice.
• A determinate number of threads could be spun up to access the array buffer(queue) to make sense of the data, update a sensor related object graph and ultimately free memory.

Note:

• Converting the values to CSV appears to be an unnecessary step, aside from obvious memory allocation impact for String values, it adds additional process overhead for no apparent benefit.
• .Net (i.d. VB.Net or C#) is not an ideal choice as it uses GC (garbage collection), meaning you have no direct control over when memory is actually freed, or potentially more problematically how long your threads will be paused during a GC cycle. FYI the same would apply to Java.
• C or C++ are far better suited to this task, both require manual memory management, which initially might seem like a lot of work, but in the long run both will ensure the most consistency and smallest memory allocation.
• I also think you could also consider doing a lot more of this on the Arduino

 

[SBSP]

The solution appears fairly obvious.

You need to read up on multithreading. For example:

• 1 thread would focus solely on receiving the data and storing this in memory (array buffer), queue structures seem to be an obvious choice.
• A determinate number of threads could be spun up to access the array buffer(queue) to make sense of the data, update a sensor related object graph and ultimately free memory.

Note:

• Converting the values to CSV appears to be an unnecessary step, aside from obvious memory allocation impact for String values, it adds additional process overhead for no apparent benefit.
• .Net (i.d. VB.Net or C#) is not an ideal choice as it uses GC (garbage collection), meaning you have no direct control over when memory is actually freed, or potentially more problematically how long your threads will be paused during a GC cycle. FYI the same would apply to Java.
• C or C++ are far better suited to this task, both require manual memory management, which initially might seem like a lot of work, but in the long run both will ensure the most consistency and smallest memory allocation.
• I also think you could also consider doing a lot more of this on the Arduino

Thanks that’s advice taken, Will give C++ a go , But It might take me a long time to figure things out.

What do you mean by

I also think you could also consider doing a lot more of this on the Arduino

?

As far as the arduino goes , I can create an "Interrupt" to make the arduino wait until the PC confirmed it read the data, but whilst that happens the CAN-BUS is still independently communicating.

 

[[)roi(]]

Thanks that’s advice taken, Will give C++ a go , But It might take me a long time to figure things out.

What do you mean by ?

As far as the arduino goes , I can create an "Interrupt" to make the arduino wait until the PC confirmed it read the data, but whilst that happens the CAN-BUS is still independently communicating.

If you've never coded in either C or C++; you might find C a lot easier to pick up: it's a simple procedural style programming style (simple functions and data types), meaning you won't be bogged down e.g. with the C++ complexities like Object Oriented Programming and Multiple Inheritance, etc...

Basically you can also write additional C/C++ code to run directly on the Arduino i.e. To reduce or cutout the amount code required on your PC; the practicality of this will depend on the code process complexity and the speed of the Arduino.

Btw just to clarify: I'm not saying you can't make this work in VB.Net, it could be possible. The only reason I mentioned C/C++ is because with both of those you have far more control over resources, hence they're the only practical choice for developing time and resource sensitive code e.g. device drivers

 

[halfmoonforever]

The adruino can always send data and wait for an ack?

 

[animal531]

C# is pretty fast, GC or not. Either way the app (even with VB.NET) should be a lot faster than the serial connection. You may not use C# for a device driver, but for a serial data processing app it's more than enough.

There are a lot of easy ways to process the data, for example create threads to read new data into a byte array and hand the data off to a processing thread.

 

[Magnum]

Interesting read...

The programmable link process has fuses connecting all possible links between the internal AND gates. Programming it causes a current to pass thru the selected link and to blow the fuse. That is why once incorrectly programmed the controller is thrown away as the fuses cannot be repaired, and only the un-programmed links remain forming the programmed Chip.

Anti-fuse process works in the opposite way. you start with two connector pads with a thin isolating film in between. Programming them then burns the isolator away and causes a short.

 

[[)roi(]]

C# is pretty fast, GC or not. Either way the app (even with VB.NET) should be a lot faster than the serial connection. You may not use C# for a device driver, but for a serial data processing app it's more than enough.

There are a lot of easy ways to process the data, for example create threads to read new data into a byte array and hand the data off to a processing thread.

Pretty fast is debatable; in my experience C# is generally slower by a factor of 2 or more. It's never clear cut & a number of factors can throw this off, for example:

• Speed of the can-bus and/or including event frequency
• Event decoding and processing: higher workload = higher potential RAM / CPU usage
• Available RAM / CPU cores: less naturally equates to more GC pauses & higher potential for dropped events.

The difference as you probably know between C / C++ and .Net is that the former allows direct control over resource meaning performance can be guaranteed even on limited resource hardware e.g. an Arduino, secondly the overhead of .Net itself; the former can allow for offloading the process to the Arduino or the ultra tiny chip based computers, not so with .Net (which is beyond the typical capacity of these)

 

[gkm]

I think a buffer string is bad news in this case. As the string get longer, it starts getting really expensive to add more data at the end and to chop data from the front. And as the string gets longer, more CPU has to be spent on it, making you fall even farther behind, causing a downward spiral. What I would suggest you do is read the serial from the < to the > and put that string into a list or queue or something, then start reading again for the next string from the < to the >. Then in another thread process the strings from the queue or whatever you use. Make sure that you use a threadsafe data structure to share between the threads. This way you always only deal with short strings. Long strings are CPU killers.

Also, watch our for traversing a growing list on every message to check if it should be ignored. That also get slower as the list gets longer. You rather want to use some kind of data structure that has fast access even with a growing set. Something like a HashSet is probably a better bet. Some background reading on this can be found for example over here: http://cc.davelozinski.com/c-sharp/fastest-collection-for-string-lookups and a more general discussion of what is killing your CPU: https://en.wikipedia.org/wiki/Analysis_of_algorithms

Maybe if you optimize your code by avoiding the big buffer and using the HashSet's etc, you might not even need the two threads and can process the items as you read them.

In general I think it is more important to use correct techniques in this case, rather than a specific programming language, so I would ignore the debates about VB/C#/C/C++ and rather focus on the algorithm fundamentals. Things like long buffer strings will break the CPU in any language. Even through in general it might be a good idea to move on from VB, I do not think it is directly relevant to this problem.

This article might also give you extra inspiration: http://martinfowler.com/articles/lmax.html

 

[SBSP]

Instead of creating one big buffer and keeping track using a cursor position, I'm only creating a one line buffer that's about 40 characters long. I then split the message into 9 different messages using a "," as a delimiter.

And it seems to be quite quick, I added a few routines to group the different Message IDs together.
I wanted to post the code but the forum blocks it, It thinks I'm trying to inject SQL code

Previously when I did the above more than often it would have an incomplete message in the buffer,
I.E

<1122,22,33

or

44,44,66,77>

where its supposed to be

<ID,BIT1,BIT2,BIT3,BIT4,BIT5,BIT6,BIT7,BIT8,>

I think its because at that very moment thats the only info there was to receive from the serial port, and if I increased the delay on the Arduino it became more accurate, That was when I was still using a Laptop, I now have a i7-4790K CPU Desktop PC, So I'm not too sure if the above solved the issue.



Below is a 1Min Video showing it.
[video=youtube_share;nNIDldeCJR8]https://youtu.be/nNIDldeCJR8[/video]

 

[[)roi(]]

Put your code on github.com, either as a full project or as a gist; gists are simple, you can simply copy / paste. From what understand you're retrieving the data as an array of unassigned char, for example:

unsigned char buf[8]

Which you then output as a comma delimited string prefixed/suffixed with

I.e. Your current operation on the Arduino is nothing more than CSV prep, and that's where I think you should rather consider doing a bit more, for example:

1. Write code to make sense of the data, and then present that data as a JSON web service, or even a formatted web page or both. This implies the Arduino will run a web server (reference: https://www.arduino.cc/en/Tutorial/WebServer)
2. Alternatively just use bit shifting to pack the unassigned char data into a unassigned uint32_t or uint64_t -- i.e. Reduce the overhead of the string
3. just send the character value for each unsigned char

The benefit for option 1 should hopefully be clear; you're basically setting up a solution where the Arduino does the work and the PC can be optional. In this case you build your code to run on the Arduino, which means the more obvious language choice is C. Your current Arduino CSV operation is already C.

Option 2 only really lightens transmission over the CSV version + numerical operations are substantially faster than string, including using less RAM and CPU. Bit packing will reduce the amount of characters sent serially to the PC, e.g. Using uint32_t 1 line value will be transmitted as 5 unicode characters (1 for ID and 4 for the eight packed values). Btw unpacking this on the PC is easy.

Option 3 relays the data as 9 characters (ID + 8) -- unsigned char is 8 bits, or decimal 0 to 255, meaning the character value will be in the extended ASCII range (conversion is simple); like option 2 it substantially reduces the data transmitted serially versus CSV string & like option 2 no comma is required re you know each sequence is 9 characters

Code example for option 3
https://gist.github.com/AfricanSwift/b772917d38446880bf13a0aad2f3dd32

In VB you'll then use Encoding.Getbytes() method to convert the characters to integer values.

The entire process could be multithreaded, for example:

• one thread deals with the serial communications, taking off groups of 9 characters and dropping this onto queues associated with the decoding threads, round-robin scheduling approach.
• Decoding threads, use Getbytes to retrieve values and pass this along (via queue) to a thread that updates the Object Graph
• ultimately you want the main thread to pick up on UI updates related to the object graph.

Naturally you need to consider the best way to design your object graph for this data, covering updates, history, caching, etc...

 

[gkm]

I think its because at that very moment thats the only info there was to receive from the serial port, and if I increased the delay on the Arduino it became more accurate, That was when I was still using a Laptop, I now have a i7-4790K CPU Desktop PC, So I'm not too sure if the above solved the issue.

Yes, you probably want to keep on read one character at a time until you get your "end of message byte", which in this case is the ">" and pack the characters into the buffer as you read them. Then when you have your whole message, process it, then start reading again for the next message, checking that it starts with a "<". If it does not start with a "<", then you discard characters until you get to the "<" as a re-sync. Maybe log when you are discarding, so that if it happens a lot then you know you have a problem that needs investigation.

Serial data is a stream, so you cannot assume it will arrive in nicely defined chunks. Seems like you are fairly lucky so far in terms of the neat chunks arriving, but you cannot assume it will always be like that and get reliable results.

 

[[)roi(]]

Serial data is a stream, so you cannot assume it will arrive in nicely defined chunks.

canbus response size is clearly defined?

INT8U MCP_CAN::readMsgBuf(INT8U *len, INT8U buf[])