Add deref value types: large values out of index keys

[maxweber]

dbval encodes each datom — value included — into every index key (eavt/aevt/teav, plus avet when indexed). That breaks on SlateDB, which caps keys at 64 KiB, and it makes scans expensive for large values: every scanned datom edn/read-strings the full value even when a filter discards it. Storrito's FabricJS model EDN strings regularly reach several hundred KiB.

Deref value types

Attributes flagged with {:dbval/deref true} store only a 32-byte SHA-256 of the value's pr-str UTF-8 bytes in the v-position of their index keys. The pr-str bytes themselves are stored once, content-addressed, in a new blob area of the store:

• SQLite: dbval_blob (h blob primary key, v blob) WITHOUT ROWID, written in the same transaction as the keys
• SlateDB: packed ("blob", <hash>) keys with the bytes in the KV value (values may be up to 4 GiB), in the same WriteBatch
• memory: a second skip-list map

ITupleStore accordingly gains a blobs argument on -commit! and a new -get-blob.

Reads surface a BlobRef, a clojure.lang.IDeref/IPending: the value is only fetched and parsed on @, cached per instance. Printing shows #dbval/blob-ref "<hex>" and never fetches, so logging query results stays safe.

Why a content hash (not a random blob id)

serialize-value must remain a pure function of the value: fsearch db [e a v] (no-op detection, retract-by-value, cardinality-many), the upsert/:db/unique AVET scans, and the datom= post-filter all reconstruct the v-component from a plain value. With a content hash, all those engine paths work unchanged — equality is hash equality (same content ⇔ same hash), so re-asserting an unchanged value is detected as redundant without fetching anything, :db/unique and upserts keep working, and datalog equality joins compare 32-byte hashes. Content-addressing also makes blob writes idempotent and deduplicating.

Known caveat (documented in the code): Clojure printing is not canonical — two = small maps built in different insertion orders can pr-str differently and get different hashes. Hash equality is therefore conservative: never false-equal, worst case an occasional redundant retract+assert.

Details

• Search patterns normalize a bound v to a BlobRef (resolve-pattern-v), so scan ranges and the datom= filter both compare hashes.
• A pending blob overlay (pending-blobs on DB, mirroring the pending key overlay) is committed atomically with the keys and gives transaction functions and chained dry-runs read-your-writes on values asserted earlier in the same transaction.
• Legacy inline datoms — written before their attribute was flagged — read as BlobRefs that hash the inline string, so equality against blob-backed datoms of the same value keeps working. They re-serialize to their original inline form (inline-str), which keeps retraction keys adjacent to the assertion keys they cancel out. No data migration needed; find-exact-datom also finds them where the hash-ranged search cannot.
• {:dbval/deref true} validates against :db/valueType/:db/tupleAttrs combinations; :db/unique and :db/index remain supported.
• Values that serialize to more than 60 KB on a non-deref attribute now throw :transact/value-too-large with a message pointing at the flag, instead of failing with an opaque native SlateDB error mid-commit.

Testing

• Full JVM suite: 185 tests, 1120 assertions, green.
• New dbval.test.deref-value (13 tests): round-trip via query/entity/pull, laziness, print form, no-op re-assert, retract+assert on update, history, retract by value / by BlobRef / retractEntity, copy-without-fetch, equality join, unique upsert + lookup ref, cardinality-many, CAS, tx-fn read-your-writes, chained dry-runs, legacy inline datoms, oversize guard, SQLite reopen persistence.
• dbval.store.slatedb-test gains a test that round-trips a ~300 KiB value through SlateDB — impossible before this change — and checks that blob keys never leak into index scans.

🤖 Generated with Claude Code