Separate EBML from Matroska, add another EBML-based format: MOSAIC - #1375
Conversation
This will ease the parsing of other EBML-based formats. It also checks the EBML doctype, so matroska will not probe positively on any EBML file.
This flags elements that don't need to be decoded to an array because, according to the schema, they should appear only once in their parent element.
wader
left a comment
wader
left a comment
There was a problem hiding this comment.
Overall looks good i think 👍 had some comments and thoughts
| interp.RegisterFS(mosaicFS) | ||
| } | ||
|
|
||
| func decodeMaster(d *decode.D, bitsLimit int64, elm *ebml.Master) { |
There was a problem hiding this comment.
This is much nicer and produces much nicer decode tree than the matroska code. I'm not sure why i didn't opt for a similar structure, maybe was a bit unsure about uniqueness restrictions in ebml (or was spooked by https://www.rfc-editor.org/rfc/rfc8794.html#name-identically-recurring-eleme, but not sure i've even seen it in practice?). Will probably convert matroska later to something similar, i did a quick hack using your code it seems to work.
There was a problem hiding this comment.
indeed matroska has a few elements having maxOccurs="1" and recurring="1", so those could not be rendered that way - maybe they should not be flagged "singleton" by the current code generator?
also that works nicely (espacially for arrays) as long as elements are coherent, maybe that happens less in matroska?
There was a problem hiding this comment.
Ah didn't notice the recurring attribute in the schema, that is good to know. Not sure how often it happens in matroska, will do some research later on. But yes sound reasonable it should not be a singleton in those cases, maybe could replace singleton with maxOccurs and recurring and let format decoder code decide what is a singleton etc?
There was a problem hiding this comment.
that's a good idea, the code generator should just copy such XML attributes. Changing this.
wader
left a comment
wader
left a comment
There was a problem hiding this comment.
This looks mergable to me! had one comment about singleton but that can be done later if it make sense. Anything else you want to add or change?
|
If you have asciidoctor installed you could do a |
This avoids relying on a mosaic-specific isSingleton(). By the way: complete elements and attributes' coverage, fix a documentation string.
|
I've completed and fix and few EBML elements by the way. It required a minor modification of Matroska's tests. I think we're good ! |
| } | ||
| if c.MaxVer != "" { | ||
| fmt.Printf(" MaxVer: %s,\n", c.MaxVer) | ||
| } |
There was a problem hiding this comment.
Ugg the code for this tool is getting quite ugly 😬 but not sure it's worth tidying up, maybe later :)
|
Very nice, thanks and hope fq will be useful! what are the main use cases? used while developing or debugging corruption etc? |
|
Debugging corruption while developing, at least. We also care to be readable outside of the archive's infrastructure, so a CLI tool with a query language is an interesting alternative to our python/rust API. Although that supposes fq scales to our typical files' size: in the order of 100GB. I'll test that next week. |
|
👍 About file size: fq's resource usage should mostly scale by number of decode values and how much memory their actual/symbolic and descriptions uses. The bits "backing" a decode value is only keept track of as a range and are read on-demand if needed. Also maybe good to know that "raw" fields (ex But note that i haven't spend that much time trying to optimize memory usage so ideas are welcome! |
2 participants
as discussed in #1371
the only modification to the Matroska parser is a check of the
doctypefield; without this, Matroska would probe positively any EBML file.