CAN Data Logger
Version 1 of the CAN Data Logger: Raspberry Pi + SK Pang CAN Bus Shield + Adafruit RTC
Overview
Close to the start of my work with Prairie Machine & Parts (an electric mining vehicle manufacturer), I was asked to create a data logger that would record various operating parameters on a vehicle. At the time, we were facing the problem of having the occasional vehicle break down underground, but not knowing why. Asking the miners questions about how they drive the vehicle, always resulted in the implausible "slowly and carefully" reply. A data logging tool would help us characterize how our vehicles are being used, and keep an operating log for diagnostic purposes.
The requirements were:
After developing the tool for a few months there was also a discussion about using the same hardware to create a diagnostic tool.
The added requirements were as follows:
Together, this became the Service Tool that I developed. The source code for the Service Tool can be found here (data logging & diagnostics) and here (web-app). I really enjoyed working on this project, taking it from conception to a functioning tool that is running on tens of trucks across Canada!
Technical Implementation
The hardware used for version 1 of the Integrated Service Tool is a Raspberry Pi 2 with an SK Pang CAN Bus Shield and a Real Time Clock.
The Raspberry Pi is a computer-on-a-chip - it can easily perform complex calculations, store large amounts of data (as much as your SD card can hold), and can be configured to run a web server. The CAN Bus shield is used to facilitate communication on the vehicle's CAN bus, and the real time clock is used to keep track of the day and time during moments when the vehicle loses all power (ex. when being worked on/maintained).
For the software, I primarily use Python as my scripting language and throw in a bit of Bash for file management through Cron (a time-based scheduler). For the web-side of things I used PHP and JavaScript. This was my first go at any sort of web development and it felt very empowering learning how to create an interactive product. It's amazing how many of your questions the web is able to answer! Thank you StackOverflow.
I have a number of scripts on the Raspberry Pi that perform everything from logging the data to changing the time for daylight savings. The scripts are either called on startup, through the cron scheduler, or via the web-app. Most of the scripts are rather self explanatory if you look at the source code, but I will explain the CAN logging script in detail below.
Once again, the source code for the Service Tool can be found here (data logging & diagnostics) and here (web-app).
Retrieving data from the CAN bus:
I use a script to listen to messages being transmitted on the CAN Bus. I only care about a subset of the messages and SocketCAN has an easy method to filter them.
Close to the start of my work with Prairie Machine & Parts (an electric mining vehicle manufacturer), I was asked to create a data logger that would record various operating parameters on a vehicle. At the time, we were facing the problem of having the occasional vehicle break down underground, but not knowing why. Asking the miners questions about how they drive the vehicle, always resulted in the implausible "slowly and carefully" reply. A data logging tool would help us characterize how our vehicles are being used, and keep an operating log for diagnostic purposes.
The requirements were:
- The data logger records and stores vehicle operating parameters,
- Users can download logs,
- Logs are easy to comprehend and analyze.
After developing the tool for a few months there was also a discussion about using the same hardware to create a diagnostic tool.
The added requirements were as follows:
- Ability to read fault codes.
- Ability to look at more detailed parameters (ex. battery cell voltage, and battery cell internal resistance).
Together, this became the Service Tool that I developed. The source code for the Service Tool can be found here (data logging & diagnostics) and here (web-app). I really enjoyed working on this project, taking it from conception to a functioning tool that is running on tens of trucks across Canada!
Technical Implementation
The hardware used for version 1 of the Integrated Service Tool is a Raspberry Pi 2 with an SK Pang CAN Bus Shield and a Real Time Clock.
The Raspberry Pi is a computer-on-a-chip - it can easily perform complex calculations, store large amounts of data (as much as your SD card can hold), and can be configured to run a web server. The CAN Bus shield is used to facilitate communication on the vehicle's CAN bus, and the real time clock is used to keep track of the day and time during moments when the vehicle loses all power (ex. when being worked on/maintained).
For the software, I primarily use Python as my scripting language and throw in a bit of Bash for file management through Cron (a time-based scheduler). For the web-side of things I used PHP and JavaScript. This was my first go at any sort of web development and it felt very empowering learning how to create an interactive product. It's amazing how many of your questions the web is able to answer! Thank you StackOverflow.
I have a number of scripts on the Raspberry Pi that perform everything from logging the data to changing the time for daylight savings. The scripts are either called on startup, through the cron scheduler, or via the web-app. Most of the scripts are rather self explanatory if you look at the source code, but I will explain the CAN logging script in detail below.
Once again, the source code for the Service Tool can be found here (data logging & diagnostics) and here (web-app).
Retrieving data from the CAN bus:
I use a script to listen to messages being transmitted on the CAN Bus. I only care about a subset of the messages and SocketCAN has an easy method to filter them.
candump -t A -n 10 can0,123:7ff,456:7ff
Where 123 and 456 are the message IDs that I wish to retrieve. I will ask for 10 messages at a time (that's the number after the n), parse them, and then ask for more. EDIT: Since creating this Service Tool, there have been updates to the CAN Shield and SocketCAN; anyone wishing to implement a CAN data logger should follow the examples provided by SK Pang on their Github page.
After retrieving the CAN messages, I will take the data from each message and parse it. Each piece of data is stored in hex, and in little endian byte order. As well, some data is signed, some is unsigned, and most data has to be multiplied by a factor to get the actual value.
After the parsing is complete, I will store the data in a database. I used SQLite3 as my choice of database, as it is lightweight but is nonetheless able to perform to my standards (I really don't ask much of it). The CAN messages come in a lot faster than I care to record, so I only record data once every second.
Lastly, I also record data to another table in the database called the summary table. Here I sum some of the data over time to calculate the total distance traveled by the vehicle, energy consumed, and other parameters.
The CAN logging script described above is called on startup (using /etc/rc.local). However, just in case the process didn't start correctly, or encountered an error - I also use a bash script, called via Cron, that checks if the process is running and calls the script if it was not.
Displaying data to the user:
Now, you may wonder - the data is in the database, but how does a user access it? For this I have created a front-end where a user may download the summary log or their selection of daily logs. The screenshots below display the Integrated Service Tool user-interface. Before you jump to judge me, please keep in mind that this is my first ever web-development project - and I learned everything I implemented via Coursera courses and countless forums.
After retrieving the CAN messages, I will take the data from each message and parse it. Each piece of data is stored in hex, and in little endian byte order. As well, some data is signed, some is unsigned, and most data has to be multiplied by a factor to get the actual value.
After the parsing is complete, I will store the data in a database. I used SQLite3 as my choice of database, as it is lightweight but is nonetheless able to perform to my standards (I really don't ask much of it). The CAN messages come in a lot faster than I care to record, so I only record data once every second.
Lastly, I also record data to another table in the database called the summary table. Here I sum some of the data over time to calculate the total distance traveled by the vehicle, energy consumed, and other parameters.
The CAN logging script described above is called on startup (using /etc/rc.local). However, just in case the process didn't start correctly, or encountered an error - I also use a bash script, called via Cron, that checks if the process is running and calls the script if it was not.
Displaying data to the user:
Now, you may wonder - the data is in the database, but how does a user access it? For this I have created a front-end where a user may download the summary log or their selection of daily logs. The screenshots below display the Integrated Service Tool user-interface. Before you jump to judge me, please keep in mind that this is my first ever web-development project - and I learned everything I implemented via Coursera courses and countless forums.
The web-app is implemented in PHP/HTML/JavaScript.
A user can select the dates that they are interested in and press "Download Logs" to retrieve a zip file of logs. In the back, a PHP page is loaded which calls a python script,
A user can select the dates that they are interested in and press "Download Logs" to retrieve a zip file of logs. In the back, a PHP page is loaded which calls a python script,
exec("python /createZipLogs.py " . $listFiles, $output, $return);
where $listFiles is the list of parameters (days) that the user would like logs for. The Python script then asks the database for all of the information for the requested days and saves the data in .csv files. It then proceeds to zip the files and send a 'Hey I'm done, the .zip file is called <xyz.zip>' message back to the PHP script. The PHP script then produces the <xyz.zip> file for download.
The zip file produced above is only used once, as a new .zip file would be created for the user's next query. For this reason, and to preserve the SD card, this data is stored on a temporary area of the RAM disk. You can read this great tutorial about creating a RAM drive on your Raspberry Pi.
Helper scripts:
Other helper scripts that I use are:
Learning
Other things that I have learned in the process - Linux permissions can be quite the hassle when you don't realise that they are the problem! It took me some time to figure out why a Python script being called from PHP was failing, but when called through the terminal did everything I desired. As it turns out, it was trying to create a file in a directory that the user www-data (the web interface) did not have permission to write in.
All in all, I learned how to create web-interfaces, developed my python skills, and unwillingly improved my troubleshooting skills. I also learned first hand how important version control is when different trucks are shipping out the door every month. I am thankful for all of the learning opportunities that this project provided me with.
Outcomes
I am happy that the Service Tool has been rolled into production and now ships on every truck that PMP produces. The Service Tool has already pleased some customers, and spiked the interest of others. It has also been helpful in diagnosing some problems on remote trucks. For example, from amperage-draw data, you can tell when certain features (ex. AC, or battery blankets) are being used or if the truck is driving up-hill or down-hill, which is helpful in diagnosing when certain problems occur or what can be causing them.
I am sure that the Service Tool will be useful to Prairie Machine and Parts as they expand into ever further-away markets. I understand that since my leaving, they have hired a dedicated developed to continue working on the Service Tool.
The zip file produced above is only used once, as a new .zip file would be created for the user's next query. For this reason, and to preserve the SD card, this data is stored on a temporary area of the RAM disk. You can read this great tutorial about creating a RAM drive on your Raspberry Pi.
Helper scripts:
Other helper scripts that I use are:
- cleanDBs.py - a script that checks how much space is left on the disk, and if it is above a certain threshold (95%) starts to delete logs, day by day, from the database.
- cleanRAM - a script that cleans up files from the RAM 'drive' 4 minutes after they were made.
- daylight_savings.py - a script that compares today's date against dates when the clock needs to be set forward or set back by an hour and does so if necessary.
- turn_on_led.py - used to set one of the RPi's GPIO pins to HIGH, to turn on an LED. I use this as the last line of /etc/rc.local to signify that the Pi has booted correctly. This is useful as the Pi is all packaged up in a weather-proof case, and a user would have no other way to debug the Pi.
Learning
Other things that I have learned in the process - Linux permissions can be quite the hassle when you don't realise that they are the problem! It took me some time to figure out why a Python script being called from PHP was failing, but when called through the terminal did everything I desired. As it turns out, it was trying to create a file in a directory that the user www-data (the web interface) did not have permission to write in.
All in all, I learned how to create web-interfaces, developed my python skills, and unwillingly improved my troubleshooting skills. I also learned first hand how important version control is when different trucks are shipping out the door every month. I am thankful for all of the learning opportunities that this project provided me with.
Outcomes
I am happy that the Service Tool has been rolled into production and now ships on every truck that PMP produces. The Service Tool has already pleased some customers, and spiked the interest of others. It has also been helpful in diagnosing some problems on remote trucks. For example, from amperage-draw data, you can tell when certain features (ex. AC, or battery blankets) are being used or if the truck is driving up-hill or down-hill, which is helpful in diagnosing when certain problems occur or what can be causing them.
I am sure that the Service Tool will be useful to Prairie Machine and Parts as they expand into ever further-away markets. I understand that since my leaving, they have hired a dedicated developed to continue working on the Service Tool.