-
Notifications
You must be signed in to change notification settings - Fork 2
Add UDP packet format docs #205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 3 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| # Datastreaming: UDP packet formats | ||
|
|
||
| This page describes the UDP packet format for instruments which stream UDP from detectors (e.g. HRPD-X). | ||
|
|
||
| Translation between this UDP format and the {doc}`flatbuffers format <ADRs/000_kafka>` is done by [`event_udp_to_kafka`](https://github.com/ISISComputingGroup/event_udp_to_kafka), which initially get written to the `_rawEvents` Kafka topic. | ||
|
|
||
| The data comes in over UDP packets, each of which is made up of 32-bit words. All data is transmitted in big-endian | ||
| format. | ||
|
Comment on lines
+7
to
+8
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can/will the UDP packets have trailing padding bytes? |
||
|
|
||
| ## Header format | ||
|
|
||
| The header is 16 words (64 bytes). | ||
|
|
||
| ### Word 0: marker byte | ||
|
|
||
| Always `0xFFFFFFFF`. | ||
|
|
||
| ### Word 1: header information | ||
|
|
||
| - **Bit 0**: End of run header marker (active low) | ||
| - **Bit 1**: Veto frame packet header marker (active low) | ||
| - **Bit 2**: Sample environment packet header marker (active low) | ||
| - **Bit 3**: Not used | ||
| - **Bits 4..=31**: Always `0xFFFFFFF` | ||
|
Comment on lines
+20
to
+24
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If all bits are high is that a neutron data packet? |
||
|
|
||
| ### Word 2: Information | ||
|
|
||
| - **Bits 0..=3**: Reserved for header type | ||
| - **Bits 4..=8**: Length of header | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bytes or words? |
||
| - **Bits 9..=10**: Not used | ||
| - **Bit 11**: Bad frame | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What exactly constitutes a "bad frame"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was what that bit was used for in DAE2/3. Not sure if it was actually implemented on those designs.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we need a version number in the header for if we do change layout? Or can we be sure we will always be able to update all possible clients and not need to read any old data?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. or that a board with old firmware gets used and confuses/crashes new clients There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think having a header type makes sense if it helps provides some flexibility in the future.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Isn't word 2 bits 4..=8 already "length of header"? We could almost certainly accomodate different-length headers later if they had e.g. a different "header information" word (maybe later we have a 'EndOfRunHeaderV2' header type)?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess i had a bit of a general design question here in that should we have a standard ISIS UDP header format that we use for anything we might do, be it detector streaming or something else in future, with the idea to be extensible and avoid the problem of some software misinterpreting a UDP packet. Maybe word 0 (marker byte) could do this to some level. Currently word 0 is fixed, word 1 is some detector specific flags, word 2 has a header type (4 bits) plus some other detector/dae specific stuff. It feels header type should be earlier on in the structure as it is more general than the information in word 1 if we think of header type as not just being "detector header type" but a general packet type? |
||
| - **Bit 12**: Run will continue | ||
|
Tom-Willemsen marked this conversation as resolved.
Outdated
|
||
| - **Bit 13**: No frame sync | ||
| - **Bit 14**: Frame Memory Full veto | ||
| - **Bit 15**: Frame data overrun | ||
| - **Bits 16..=31**: Not used | ||
|
|
||
| ### Word 3: Frame number | ||
|
|
||
| - **Bits 0..=31**: frame number as u32 | ||
|
|
||
| ### Word 4: GPS timestamp | ||
|
|
||
| - **Bits 0..=3**: seconds (most significant bits; combine with least significant bits from word 5) | ||
| - **Bits 4..=9**: minutes | ||
| - **Bits 10..=14**: hours | ||
| - **Bits 15..=23**: days | ||
| - **Bits 24..=31**: years (as offset from year 2000) | ||
|
|
||
| ### Word 5: GPS timestamp | ||
|
|
||
| - **Bits 0..=9**: nanoseconds | ||
| - **Bits 10..=19**: microseconds | ||
| - **Bits 20..=29**: milliseconds | ||
| - **Bits 30..=31**: milliseconds (least significant bits; combine with most significant bits from word 4) | ||
|
Tom-Willemsen marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Word 6: period number | ||
|
|
||
| - **Bits 0..=15**: Period number | ||
| - **Bits 16..=31**: Not used | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. in event mode period number is just an extra metadata tag - is it worth just letting it be 32 bits? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yea, just an extra metadata tag in event mode so I don't really mind either was on this. |
||
|
|
||
| ### Word 7: events in frame | ||
|
|
||
| - **Bits 0..=31**: number of neutron events in this frame. | ||
|
|
||
| :::{note} | ||
| This is not necessarily the same as the number of events in this UDP message, as the events may be split between | ||
| multiple UDP messages. This is the *total* number of events in the ISIS frame. | ||
|
|
||
| See header word 13 if looking for length of *this* message. | ||
| ::: | ||
|
|
||
| ### Word 8: protons-per-pulse | ||
|
|
||
| - **Bits 0..=7**: protons-per-pulse in this ISIS frame. | ||
| - **Bits 8..=31**: unused | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DAE3 seems to send a 10 bit PPP looking at the packet definition, but i guess we don't expect to need that amount? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The DAE3 maybe sends a 10 bit PPP but the two lowest bits are padded with 0s. If required I can pad it with 0s to make it 10 bit? |
||
|
|
||
| To convert to {math}`\mu Ah` delivered during this ISIS frame, multiply by {math}`1.738{\times}10^{-6}`. | ||
|
|
||
| ### Word 9: vetoes | ||
|
|
||
| - **Bit 0**: FIFO veto | ||
| - **Bit 1**: SMP veto | ||
| - **Bit 2**: TS2 pulse veto | ||
| - **Bit 3**: Wrong pulse veto | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What exactly is a "wrong pulse"? Is this e.g. not the first TS1 pulse when someone has selected "First TS1" timing source? |
||
| - **Bit 4**: Unused | ||
| - **Bit 5**: ISIS slow | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What exactly does this mean? |
||
| - **Bits 6..=9**: External vetoes | ||
| - **Bits 10..=13**: Fast chopper vetoes | ||
| - **Bits 14..=15**: Reserved vetoes | ||
| - **Bits 16..=23**: Unused | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there also a "isis not at 50hz" veto? (or is that "isis slow") |
||
| - **Bits 24..=31**: Frame repeat number | ||
|
|
||
| ### Word 10: next frame address | ||
|
|
||
| - **Bits 0..=31**: Address of the next frame, in bytes. | ||
|
|
||
| ### Word 11: next frame address | ||
|
|
||
| - **Bits 0..=31**: Address of the next frame, in 64-bit words. | ||
|
Tom-Willemsen marked this conversation as resolved.
|
||
|
|
||
| ### Word 12: streamed frame number | ||
|
|
||
| - **Bits 0..=31**: streamed frame number. | ||
|
|
||
| ### Word 13: packet length | ||
|
|
||
| - **Bits 0..=11**: number of 32-bit words from the beginning of this header to the start of the next header. | ||
| - **Bits 12..=31**: Unused | ||
|
|
||
| ### Word 14: unused | ||
|
|
||
| - **Bits 0..=31**: Unused | ||
|
|
||
| ### Word 15: checksum | ||
|
|
||
| - **Bits 0..=31**: Pre-DDR checksum | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Think we decided to drop this checksum at this level... |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -192,6 +192,7 @@ dbs | |
| dcalc | ||
| dcom | ||
| dcomcnfg | ||
| DDR | ||
| deadband | ||
| deadbands | ||
| deadman | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can have multiple "messages" (headers + their data) within a single UDP packet, right?