Description
Hello Kees,
I used all your logfiles for testing my code.
And I found that I need additional information for some files.
It is obvious that files like 129540.raw, 130311.raw, pgn130330.raw have been edited manually.
But some of the others need explanation from you.
I implemented a parser into HUD ECU Hacker for all logfile formats that are here on Github.
First question:
The files in EBL format follow one of the two schemes:
DLE STX <command> <len> [<data> ...] <checksum> DLE ETX
or
ESC SOH <frametype> [<data> ...] ESC LF
The only file that does not follow this scheme is pgn1.ebl which has this content:
03 20 10 E7 A7 84 83 D9 01
1B 01 - 07 95 0E 28 9A 20 01 F8 09 3D 0D B3 22 48 32 59 0D - 1B 0A
The second line complies with the second scheme.
But the first line does not use none of the schemes.
So my question is:
Is this a manually created file?
Is this file correct or incomplete?
Are there 4 bytes missing in the first line?
Is it allowed that a sequence of bytes appears without enclosing
1B 01 ..... 1B 0A
or
10 02 ..... 10 03?
I suppose that the file is corrupt because EBL Reader shows the time in red:
Logfiles are a very important source for testing.
They are even indispensable.
It would be really helpful if you would upload a file Readme.txt that explains for each logfile:
1.) The hardware that has recorded this file,
2.) The software that has recorded this file,
3.) A list of all the devices (manufacturer + model) that were connected to CAN bus,
4.) If the file has been manipulated manually.
5.) Has the logfile been created by sending artificial data to CAN bus ?
6.) Some more info about the environment, for example the real water depth to prove the correct display of the values.
There is a logfile "garmin-fake-126720.raw"
Why this name?
What is fake in this file?
An explanation would be useful.
I see several logfiles where the Longitude and Latitude point to a place where there is not even water.
For example actisense-522.candump.log has a message "Navigation Data" with coordinates: 38.4 and -2.4 deg.
If you open this in Google Maps you see that it is in the mountains in Spain inside a national park.
There is not even a city or village near to this place.
https://www.google.com/maps?q=38.4000000,-2.4000000
Apart this file contains a wrongly encoded String_LAU2 in "Route and WP Service"
So this file contains corrupt data in more than one message.
It is obvious that this is not a valid logfile from a real ship.
It is obvious that the file sample-YDWG02-2.txt has been manipulated.
The person who created this file has copied and pasted totally wrong.
Exactly identical lines are repeating and other lines, which are part of a Fastpacket, are missing.
Invalid Yacht Digital packet with more than 8 data bytes in line 76
The logfile contains inconsistent timestamps. The time is jumping backwards in line 27.
Src: 00, Logfile line: 24, 19F01400: 13 02 01 FF FF FF FF FF (Fast Packet: Start packet missing)
Src: 35, Logfile line: 52, 19F01435: A0 86 34 08 4C 02 59 44 (Fast Packet: Previous transfer not finished, 121 bytes missing)
Src: 00, Logfile line: 54, 19F01400: 13 02 01 FF FF FF FF FF (Fast Packet: Start packet missing)
Src: 1A, Logfile line: 55, 19F0141A: A0 86 34 08 00 14 57 53 (Fast Packet: Previous transfer not finished, 121 bytes missing)
If I open logfile: "can0-1.pcap" in Wireshark I see the time jumping back by two weeks in the middle of the file!
Later it jumps two weeks forward to the previous time at packet Nº 1405.
The only explanation that I can imagine is that someone has manipulated this file.
Logfile raymarine-string.raw
Here surely something is wrong.
If this is a valid logfile the definition of the 2 strings is wrong.
Which means that the fix length strings have 12 characters and not 16 as defined in the XML file.
The second string contains 2 control characters 00 and 02, so this is surely other data and does not belong to the string.
Or the logfile has been created manually and invalid data has been stored in the logfile.
It is surely not a real logfile because it contains only one message.
If we don't know where these logfiles come from and if they are real or manipulated, it is impossible to take the correct decision.
This shows once more how important it is to have more information about the logfiles.
I recommend to remove all logfiles that have been created with artificial data or that have been manipulated in any way. How do we know if the person who created the artificial file has used valid data?
I recommend to divide the logfiles into two subfolders on Github:
- one for real logfiles from real ships navigating on a real lake, river or in the sea.
- and one folder with artifical logfiles for testing with a strict warning that the content of these files may be wrong.
There are also logfile names that make no sense.
The file candump-fast-cr.txt contains only PGN 127489 "Engine Parameters"
So, what does "Fast-CR" stand for ?
There are several logfiles with corrupt time stamps.
The worst is dirona-actisense-serial.raw
It has 723 errors.
When I load it with my logfile parser I get:
The logfile contains inconsistent timestamps. The time is jumping backwards in line 171.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 187.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 190.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 202.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 520.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 523.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 526.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 527.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 547.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 797.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 800.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 803.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 804.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1108.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1130.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1136.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1138.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1145.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1407.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1412.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1414.
The logfile contains inconsistent timestamps. The time is jumping backwards in line 1430.
etc...
And when I take a look into the file I find several syntax errors:
Suddenly the milliseconds are missing and the time is jumping.
Was this recorded by a low quality software?
Or has this file been manipulated manually?
Here the time is jumping back 4 seconds and then it jumps forward even SEVEN seconds.
And the milliseconds are missing again.
This is a logfile of a very low quality.
What software has grabbed this?
These errors are also in several other logfiles.
Here I fixed the logfile Merrimac 2022.
I sorted all lines by timestamp to analyze in Araxis Merge what is going on.
Strangely all the Request messages (PGN 59904) come approx 40 ms from the future.
How is this possible ?
The logfile raymarine-ev1.raw has another problem: The timestamps are stuck.
This logfile tells that during one millisecond (the millisecond 787)
three hundred fourty five messages have been received!
This is completely impossible.
Here you see a real capture from my oscilloscope transferred to the computer over USB
and analyzed with my open source software Oszi Waveform Analyzer:
https://netcult.ch/elmue/Oszi-Waveform-Analyzer
On a CAN bus with 250 kBaud a packet with 29 bit ID, DLC, 8 data bytes, CRC and all the stuffing and special bits takes 560 µs.
That means that not even 2 packets fit entirely into one millisecond.
But Raymarine tells us that 345 packets have been transferred in one millisecond.
Why do I need logfiles of good quality ?
I did a very deep analysis of all the NMEA PGN's.
I compared:
1.) Your Canboat.XML file
2.) The file NMEA 2000 PGN Appendix B (2009).pdf
3.) What Actisense NMEA Reader (latest version) displays.
I could write an entire book about all the errors that I found.
All this NMEA stuff is full of bugs.
The documents from the NMEA have lots of obvious errors which they never fixed.
Here only one example from many:
This file is a Corrigendum of the Heartbeat message.
And this corrigendum contains 3 contradictions.
1.)
Parameter 1 is named "Update Rate" which makes sense for a heartbeat message.
But the comment beside talks about an "Offset" which makes no sense.
We have a contradiction.
What is correct ?
Is it an interval or an offset ?
You have renamed "Update Rate" into "Data transmit offset". This is surely wrong.
The text above talks about a transmission "interval" twice.
I would clearly say that the name is correct and the comment is wrong.
Probably they copy and pasted this comment from another message.
An offset makes no sense for a heartbeat. Offset from what?
2.)
This comment also contradicts the explanation above.
Above it says: "This PGN cannot be requested"
The comment says: "Offset in transmit time from the request command"
So what is correct?
Can it be requested or not?
Once again I would clearly say that the comment is wrong.
What sense would it make to REQUEST a heartbeat message?
This would mean that the patient is already dead.
The heart is not beating anymore and needs a "request" to do one beat and then it stops beating again until the next request?
What a nonsense.
Obviously a heartbeat is a message the beats from alone without needing a request.
Otherwise the name "Heartbeat" would be wrong.
And there is parameter 2 "Heartbeat Sequence Counter" which would also make no sense if the message would not beat from alone.
3.)
But there is even a third contradiction in this so called "corrigendum".
The explanation above says that the transmission interval can be between 1 ms and 60 seconds.
But below it shows a range from 0 to 655 seconds.
So which range is correct?
1 ms to 60 seconds
or
10 ms to 655 seconds ?
The Resolution says "1x10E-2 seconds" which would mean a division by 100 of the raw value.
But the explanation above would result in a division by 1000.
As the NMEA is not able to give us useful documents we have to GUESS what might be their intention.
Or may be not even the NMEA knows what is correct?
How will companies like Maretron, B&G, Navico or whoever implement the heartbeat correctly with such a sloppy documentation?
I would say that a range from 10 ms to 655 seconds makes more sense than 1 ms to 60 seconds.
Who will need a hearbeat every millisecond?
On a CAN bus of 250 kBaud this would produce a tremendous amount of useless traffic.
Even 10 ms is an absurdly short interval for a heartbeat.
To investigate this, I searched in the logfiles for heartbeat messages.
I thought it would be easy to find out if the factor is 100 or 1000 by studying the frequency of the heartbeat messages.
There are 2 logfiles which contain PGN 126993:
1.) raymarine-ev1.raw
2.) susteranna2020.raw
All heartbeat messages have a raw value of 60000 (0xEA60) for the interval.
So if the correct Resolution is 1000 I should see this message every 1 minute.
And if the correct Resolution is 100 I should see this message every 10 minutes.
The Raymarine file has heartbeat messages, for example for device 172:
Timestamp CAN ID + DLC Interv Counter
2024-02-17T05:15:49.776Z, 7,126993,172,255,8, 60,ea, 13, ff,ff,ff,ff,ff
2024-02-17T05:15:49.790Z, 7,126993,172,255,8, 60,ea, 14, ff,ff,ff,ff,ff
2024-02-17T05:15:49.800Z, 7,126993,172,255,8, 60,ea, 15, ff,ff,ff,ff,ff
2024-02-17T05:15:49.811Z, 7,126993,172,255,8, 60,ea, 16, ff,ff,ff,ff,ff
2024-02-17T05:15:49.821Z, 7,126993,172,255,8, 60,ea, 17, ff,ff,ff,ff,ff
This logfile shows 2 things:
1.) The messages are intact because the sequence counter increments correctly: 13, 14, 15, 16, 17
2.) There is no request message for a heartbeat in the entire file of 1,7 Megabyte.
But this file is useless to determine the interval because the timestamps are garbage.
Between one heartbeat and the next there are nearly 6000 other messages.
The buggy timestamp advances only 10 ms for 6000 messages!
Also the file Susteranna has multiple heartbeat messages.
But for each device there is only one heartbeat.
So an interval cannot be determined.
SUMMARY:
The quality of the logfiles does not allow me to analyze the frequency of the heartbeat message.
I see that a lot of hardware / software that you are using here to create these logfiles is of worst quality.
You are lucky that soon my next version of HUD ECU Hacker 5.5 will be released which is high quality software.
I wrote 11000 lines of code to implement the J1939 and NMEA 2000 protocols entirely.
I fixed the errors in the databases that can be fixed, so you will see correct values.
And obviously you will see correct timestamps in the logfiles.
And you can use a J2534 adapter for only $20 US which does not lose packets because it works at Full USB speed.
https://netcult.ch/elmue/hud%20ecu%20hacker/index.php#NMEA_2000
I also made a deep analysis of Actisense NMEA Reader.
I found many bugs.
1.)
Actisense did not implement the majority of the lookup values.
They simply show a useless integer number instead of the meaning.
For example: "AIS Reporting Interval" may be displayed as a useless value "5" in Actisense.
What does "5" mean ?
If they would have implemented the lookup value for DD302 they would display "30 seconds" instead.
All the PGNs that they implement contain totally 194 lookup values.
Actisense implements only 54 lookup values that their software will decode.
All the remaining 140 lookup values are displayed as useless integer numbers.
They did a very lazy work.
2.)
The 64 bit NAME fields are not decoded.
3.)
Not even such a basic thing as decoding Unicode String_LAU is implemented.
4.)
They use a wrong repeat field count for
PGN 130573 'Entertainment - Supported Source Data'
The result is that from the second audio/video source on you see crippled data.
5.)
They did not even implement the repeating fields at all for
PGN 126464 'Transmit PGNs group function'
PGN 130571 'Entertainment - Library Data Group'
PGN 130573 'Entertainment - Supported Source Data'
PGN 130574 'Entertainment - Supported Zone Data'
PGN 130583 'Entertainment - Available Audio EQ Presets'
PGN 130584 'Entertainment - Bluetooth Devices'
In all these messages multiple repeating blocks of fields can be sent.
You will only see the first one.
11.)
They did not implement String_LAU2. You will see crippled strings in
PGN 130070 "Route and WP Service - WP Comment"
PGN 130071 "Route and WP Service - Route Comment"
PGN 130072 "Route and WP Service - Database Comment"
14.)
They use a wrong factor for the parameters DD052, DD149, DD200.
You will see wrong distance / pressure in
PGN 130066 'Route and WP Service - Route/WP-List Attributes'
PGN 130069 'Route and WP Service - XTE Limit & Navigation Method'
PGN 130073 'Route and WP Service - Radius of Turn'
PGN 127237 'Heading/Track Control'
PGN 130324 'Moored Buoy Station Data'
19.)
Not even the important Request / Command / Acknowledge Group Function is implemented.
Actisense is not able to display the source address correctly.
Also the Acknowledge Group Function is decoded totally wrong.
The next screenshot shows a field "Fields 4 - 12 repeat as needed".
And even this is wrong, because fields 4 - 14 are repeated.
And where are they ?
And this happens when a message contains Unicode strings:
When I load a DSC Call, Actisense shows me for Value 105 in Second Telecommand "Sinking" instead of "No operator available" !
Why is Actisense selling an expensive NGT-1 adapter and betraying the clients with such a crap software?
The answer is simple:
Nobody will ever notice this because nobody has the NMEA documents and can prove if they show correct or wrong values.
The fact that the NMEA keeps all information secret allows them to sell sloppy software and their clients will even be happy because they have no chance to ever find out.
I'am the first one who did such a deep analysis.
May light come into the darkness!