Skip to content

Implement 8.3 S&F Cache VIewer#316

Open
Joshlha wants to merge 8 commits into
mainfrom
sf-83
Open

Implement 8.3 S&F Cache VIewer#316
Joshlha wants to merge 8 commits into
mainfrom
sf-83

Conversation

@Joshlha
Copy link
Copy Markdown
Collaborator

@Joshlha Joshlha commented Sep 24, 2025

This includes a few changes which I did test, though problems might creep up related to backwards compatibility (Hopefully not).

Changes

  • Changed Ignition dependencies to version 8.3.0
  • Made them transitive. (I couldn't get Gradle to download them if they weren't, for some reason)
  • Changed old S&F class alias references to use the now deprecated ones. (See StoreAndForward.kt
  • Ensured that the other things which use the Ignition SDK still work (I test the Images IDB Viewer and Alarm Cache Viewer)
  • 8.1 S&F Cache Viewer still works the same
  • Created hsql and sqlite packages under the Cache viewer tool's package

Usage

There are 3 ways you can open an 8.3 S&F Cache:

  • Open a single .idb file with the IDB Tool
  • Open a single .idb file with the S&F Tool
  • Open a .zip containing however many files with the S&F tool. (The zip opening process does not care about how the files are structured, and simply picks out all the .idb files from it)

Protobuf

  • Only two classes are properly deserialized into their Protobuf java equivalent.
  • All other classes are currently deserialized into a generic protobuf object, whose field name info is unavailable.
  • Generic protobuf objects are displayed in a JSON-like format where specific fields are just given integer values
  • Field values of classes are still viewable, so we can make some sense of the data either way.

Spotless

  • Spotless needs to be updated for it to pass on this PR. The new $$ string interpolation prefix has been added where it was prompted by the IDE.

@Joshlha Joshlha self-assigned this Sep 24, 2025
# Conflicts:
#	src/main/kotlin/io/github/inductiveautomation/kindling/cache/hsql/CacheView.kt
#	src/main/kotlin/io/github/inductiveautomation/kindling/utils/Sql.kt
#	src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt
Copy link
Copy Markdown
Member

@paul-griffith paul-griffith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some mostly minor stuff. Still hoping to find time for some exploration to see if there's a "better" way to do this, but may merge this and get it into a release ~sooner if that doesn't work out

Comment thread src/main/kotlin/io/github/inductiveautomation/kindling/cache/CacheViewer.kt Outdated
Comment thread src/main/kotlin/io/github/inductiveautomation/kindling/cache/sqlite/CacheView.kt Outdated
Comment thread src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt Outdated
Comment thread src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt Outdated
// 8.3

fun ByteArray.deserializeStoreAndForward(flavor: String): List<GeneratedMessageV3> {
val parseFunction: (ByteArray) -> GeneratedMessageV3 = when (flavor) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a better pattern to explore here where we fully instantiate the underlying S+F serdes delegates instead of relying on the "raw" proto classes, especially because if we can do that we can rely on them having a built in JSON representation format we can display and not have to generate our own toDetail impls

Copy link
Copy Markdown
Collaborator Author

@Joshlha Joshlha Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fully instantiate the underlying S+F serdes delegates

I was able to get this working with the history set, but there is no conversion logic included in the SDK to go from the Alarm Journal protobuf class to its actual normal class.

if we can do that we can rely on them having a built in JSON representation format

Even for the history set, the function to print as json was stupidly not public from what I could tell. Not a huge deal, but I had to use the protobuf JsonFormat.printer().print() on the protobuf-generated class to achieve a JSON output. It looks like the private function that prints these classes as JSON implicitly converts them to the PB-generated class first.

I'm gonna leave this one unresolved, because I just don't see a better way to do it without a bunch of work to write our own delegated classes for these PB classes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm talking about is something like this:

Index: src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt b/src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt
--- a/src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt	(revision e8c94d4709b2dd0d64a97cfac199ab3bd897d0ac)
+++ b/src/main/kotlin/io/github/inductiveautomation/kindling/utils/StoreAndForward.kt	(date 1759881879435)
@@ -6,8 +6,12 @@
 import com.inductiveautomation.ignition.gateway.alarming.journal.encoding.AlarmJournalProto
 import com.inductiveautomation.ignition.gateway.history.encoding.GenericObjectProto
 import com.inductiveautomation.ignition.gateway.history.encoding.HistoryDataProto
+import com.inductiveautomation.ignition.gateway.storeforward.data.PersistentData
 import com.inductiveautomation.ignition.gateway.storeforward.deprecated.BasicHistoricalRecord
 import com.inductiveautomation.ignition.gateway.storeforward.deprecated.ScanclassHistorySet
+import com.inductiveautomation.ignition.gateway.storeforward.flavor.PersistentFlavor
+import com.inductiveautomation.ignition.gateway.storeforward.data.history.BasicHistoricalRecord as HistoricalRecord
+import com.inductiveautomation.ignition.gateway.storeforward.serialization.DeserializationContext
 import io.github.inductiveautomation.kindling.cache.AliasingObjectInputStream
 import io.github.inductiveautomation.kindling.cache.hsql.model.AbstractDataset
 import io.github.inductiveautomation.kindling.cache.hsql.model.AlarmJournalData
@@ -122,7 +126,18 @@
 
 // 8.3
 
+object SFDeserializer : DeserializationContext {
+    private val map = mapOf<String, PersistentFlavor<*>>(
+        "history_set" to HistoricalRecord.FLAVOR,
+    )
+
+    override fun fromBytes(bytes: ByteArray, flavorKey: String): PersistentData? {
+        return map[flavorKey]?.serializer()?.fromBytes(bytes, this)
+    }
+}
+
 fun ByteArray.deserializeStoreAndForward(flavor: String): List<GeneratedMessageV3> {
+    SFDeserializer.fromBytes(this, flavor)
     val parseFunction: (ByteArray) -> GeneratedMessageV3 = when (flavor) {
         "history_set" -> {
             HistoryDataProto.HistorySetPB::parseFrom

In theory, if the individual serializers are good about being "pure" and not implicitly relying on anything from a running gateway/context, this allows us to side step all the proto related logic (except for some generic fallback behavior). This isn't fully baked and I have no idea if it works, this is just me throwing something together in literally five minutes that compiles. I don't think we have to do things this way for this PR, but I do think this is the "right" way to do things here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paul-griffith So this works for the history set, but I believe that's the only class in the SDK which has everything defined to be used like that.

It looks like in order to get this strategy working for other classes (assuming the protobuf version exists in the SDK), we'll need to write the class ourselves, and then define the flavor for it, which includes defining the serialization logic ourselves as well. It's possible but a lot of work for each class. It doesn't seem ideal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants