webPORPID has one implementation of the read-to-consensus algorithms: a C++20 core compiled as a WASI reactor. Browser workers and porpid-cli both instantiate that same public/webporpid.wasm artifact.
- A FASTQ byte stream is hashed and, for
.gzinput, decompressed incrementally. - Bounded 256-record/4 MiB batches are distributed across C++/WASM workers for filtering, primer detection, orientation, demultiplexing, and UMI extraction.
- Compact binary read records are routed by sample/UMI hash into spool partitions. With a positive
maxReadsPerSample, a monotone online cutoff prevents records that cannot survive deterministic downsampling from reaching disk. - The partitions are scanned for family counts. Sparse UMI offspring likelihoods and LDA assignments form one run-wide family model.
- Preprocessing workers are terminated. Fresh consensus workers receive only the family-model rows present in their current partition and scan that partition’s selected reads.
- Each completed encoded consensus result overwrites its consumed raw-read partition. After every partition finishes, consensus workers are terminated and the compact results are decoded one partition at a time. Only consensus and summary data continue into contamination, post-processing, MSA, FastTree, result serialization, and exports.
As accepted spool frames are committed, the same authoritative per-sample counters used for final validation are posted to the main thread for a live demultiplexing chart. Later stages report named substeps. Contamination classification, downstream annotation, collapse, and tree inference yield at bounded boundaries so progress and Skip messages remain responsive. An independent main-thread heartbeat and elapsed timer also remain animated during long consensus WASM calls.
Contamination keeps raw six-mer vectors sparse and builds bounded, temporary inverted postings over run centers and reference entries. Query dot products visit shared six-mer bins rather than rescanning every 4,096-bin vector. DP-means rebuilds this exact index after bounded center-growth increments; if the posting cap would be exceeded, the same calculation falls back to threshold-safe sparse distance pruning. The index is released with the contamination stage and is not serialized into results.
The spool is the key memory boundary. A record has a fixed 24-byte header containing its complete record length, sample index, and deterministic sampling hash, followed by UMI, read name, sequence, and quality bytes. Count and consensus passes therefore inspect the header before reading a record body.
The final downsampling cutoff for a sample is floor((2^64 - 1) * maximum / final_count). During streaming, webPORPID applies the same expression to the count seen so far. That provisional cutoff can only decrease, so it is guaranteed to retain every record that can pass the final cutoff, independent of worker completion order. Non-candidates bypass the spool, and the browser compacts stale early candidates in place every 512 MiB of admitted data and once at the final cutoff. This preserves the selected read set while preventing a highly compressed multi-million-read FASTQ from expanding into an all-read OPFS spool.
- The CLI writes each partition to an isolated temporary directory, overwrites consumed reads with compact consensus output, and removes the directory after consensus, including on errors.
- A browser analysis uses Origin Private File System synchronous access handles from its dedicated pipeline worker. Short reads and writes are completed in loops, and large inputs make a best-effort persistent-storage request. A real quota/I/O failure reports the run spool size and the browser's remaining origin quota instead of presenting a misleading truncated-frame error.
- The default external-scratch backend (where the directory-picker API is available) writes each partition once through a long-lived asynchronous stream into a temporary subdirectory on a user-selected disk. This bypasses browser-origin quota and remains optional: users can select automatic browser storage instead. Writers are closed before the count pass, partitions are then read as contiguous files, consumed read files are replaced by their encoded consensus results, and each result file is deleted immediately after decoding. The temporary subdirectory is removed after consensus or controlled failure. A tab/browser crash can leave the clearly prefixed
webporpid-scratch-*directory behind for manual removal. - If OPFS is unavailable, a bounded in-memory backend is used and stops with a clear error at 512 MiB.
Only one partition per active worker is transferred into WASM at a time. Preprocessing heaps are discarded before consensus; no worker receives the full family model; decoded reads are moved rather than copied into family groups; encoded consensus partitions are never accumulated in JavaScript memory; and all consensus WASM heaps are discarded before final assembly. The default 64 hash partitions and default 100,000-read per-sample ceiling keep working sets substantially smaller than the input dataset. Setting the ceiling to zero intentionally disables adaptive admission and retains every demultiplexed read. For very large no-downsampling runs, 256 partitions is recommended; external mode also reduces count/consensus concurrency automatically when measured partition size would create an unsafe aggregate WASM working set. A single unusually dominant UMI family still necessarily requires its reads together for faithful consensus.
A storage-free two-pass gzip mode is not used because it would not solve the grouping boundary faithfully. The first pass can determine the global UMI model, but reads belonging to one family can be interleaved from the beginning to the end of the second pass. Holding every still-open family recreates the full memory problem. Standard DEFLATE is not randomly seekable, so an offset index would require costly decompressor checkpoints and repeated block inflation; bounded partition waves would require rereading and reprocessing the entire gzip once per memory-sized wave. Direct external-disk partitioning is a single preprocessing pass, has sequential writes, preserves the existing algorithms, and is materially faster for the expected multi-million-read case.
The default worker count is the runtime's logical CPU count. The browser control and CLI --workers N option can reduce it.
Preprocessing batches can finish out of order because spool order is not semantically relevant. Family counts are merged by sample/UMI key, and reads are stably sorted by parsed input ordinal before consensus. Hash-threshold downsampling is seed-, read-, sample-, and UMI-dependent rather than scheduling-dependent. The CLI smoke suite verifies identical consensus output with one and two workers.
The consensus hot path uses sparse exact six-mer profiles, collision-free rolling 60-bit encodings for canonical 30-base seeds, contiguous buffers, and seeded long-read alignment. A seedless region uses an adaptive band whose width includes the complete length difference plus a square-root noise allowance; seeded intervals retain the original full dynamic program. Ambiguous/lower-case seed input keeps the byte-exact fallback path. The core is compiled at -O3 -msimd128 and the optimized binary contains SIMD vector operations.
Centroid profiles are counted once per read rather than allocating and rescanning a dense 4,096-bin vector. Refinement stops at its deterministic fixed point, and the final fixed-point alignments are reused for agreement counting. Agreement symbols use fixed counters instead of per-position strings. Identical families still bypass alignment without changing their sequence or agreement.
When consensus-threshold review is enabled, orchestration pauses once after consensus and any applicable contamination eligibility are known, so its abundance distribution is the exact downstream input. UMI posterior and family-size thresholds are fixed by configuration because changing them would alter which families enter consensus. The accepted consensus decision is stored in the project audit and human review time is excluded from computational timing.
A completed project can reconstruct that same consensus review from its stored family sizes and agreement values. Accepting revised values preserves UMI decisions, consensus sequences, and contamination calls and recomputes only through the highest downstream stage that was complete before the change. A global threshold change invalidates every sample. Otherwise, the effective three-threshold tuple is compared per sample and the replay atomically replaces only changed-sample records, alignments, collapse groups, trees, functional status, and edits; unchanged samples remain in place. Stages that had remained deferred or skipped are restored to that state, and append-only edit history remains.
Optional sample donor_ID values are normalized into biological self-group keys for contamination queries. Run-derived signatures from the same donor are never considered non-self; external contamination references remain non-self. The result explorer can construct an on-demand cross-sample alignment and tree for each donor while retaining sample categories and collapsed family multiplicities.
Alivibe-compatible MSA and FastTree WASM assets are packaged locally; there is no runtime dependency on Swig. Independent per-sample MSA and FastTree jobs use bounded worker pools, capped by both the configured worker count and the number of samples, so a machine with many logical CPUs does not instantiate idle runtimes.
MSA is monolithic up to 8,000 sequences and 128 MiB of input residues. Beyond either threshold, webPORPID aligns deterministic batches of 2,000 rows against a shared anchor, decomposes every batch into anchor insertion slots, merges slot widths, and reconstructs a rectangular alignment in original order. This bounds each aligner's dynamic-programming workspace while retaining the final alignment required for export and visualization.
After non-functional filtering, identical ungapped haplotypes are collapsed. Variants are ranked by decreasing member-family count (lexical haplotype tie-break) and named sample_vN_K; uncollapsed FASTA/display labels are sample_UMI fs=K minag=0.yy, while the internal join key remains sample_UMI. Each collapsed variant stores those stable member identifiers, so bubble area is proportional to family count rather than read count. Minimum agreement remains attached only to each UMI family and is not assigned to the collapsed haplotype. FastTree consumes this collapsed nucleotide alignment by default. The uncollapsed alignment is retained for optional on-demand family-level inference and minimum-agreement colouring. The browser uses the packaged double-precision BioWASM/Aioli FastTree runtime; the CLI invokes the same packaged FastTree WASM directly. Browser tree failure is logged and produces an explicit zero-branch star fallback so a completed analysis remains loadable.
The functional filter runs only on collapsed variants. It rejects sequences without a complete start-to-stop ORF, translates the remaining ORFs, and first aligns the sample variants to one another. Each returned row is checked against its exact protein before backtranslation. Symbol-only normalization is restored; a residue-count mismatch switches to an anchor/profile construction that proves every ungapped row equals its input. The translated reference is then affine-aligned to that frozen sample profile: only shared all-gap columns may be inserted into existing sample rows, so adding the reference cannot rearrange their MSA. Gaps are back-translated as codon triplets. The screening alignment is clipped to the first/last non-gap reference nucleotide to produce each variant’s trimmed coding sequence. After screening, only passing trimmed proteins are degapped and freshly aligned with bundled MAFFT 7.520 FFT-NS-2. The translated reference is added to this fixed passing-only profile using MAFFT --amino --add reference.fa passing-protein-alignment.fa, then moved to row one under its supplied FASTA name. Sample columns and every ungapped residue are verified; terminal stop symbols are preserved and gaps are backtranslated as codon triplets. This export alignment is not clipped a second time, so it retains all residues of the already-trimmed passing sequences and the reference. Filter decisions and rm annotations remain based on the screening alignment. Stored projects need collapse/functional recomputation to obtain the new export layout. Passing variants are displayed as sample_vN_K rm=0.yy, retaining identity, family count, and two-decimal match in one label. If both alignment routes fail, the failure is represented as sample-scoped state; no partial functional calls are retained and collapse/other samples continue. The alignment is retained as an editable view and can receive its own on-demand FastTree tree.
The viewer identifies the most frequent observed aligned nucleotide sequence, weighting collapsed representatives by retained UMI-family multiplicity and grouping identical uncollapsed tips on the fly. It roots the displayed tree on the branch endpoint immediately above a deterministic tip carrying that modal sequence; the modal-tip edge is exactly zero and the complete original pendant length is assigned to the opposite root edge. A midpoint-root action finds the weighted tree diameter in linear time and splits only the edge containing its midpoint. Regression tests require the complete tip set and every pairwise patristic distance to remain unchanged after either operation.
The post-consensus state is validated, MessagePack encoded, gzip compressed, and prefixed with a versioned .webporpid magic header. Contamination is an independently bypassable side gate: a deferred/skipped check produces no calls, excludes nothing, and permits requested post-processing to continue with postprocessingContaminationMode = bypassed. Post-processing remains a prerequisite for collapse, and collapse remains a prerequisite for the default collapsed tree. These states and every genuinely blocked dependent stage are stored explicitly; only a completed post-processing stage may carry its full consensus-linked record table. Small reference inputs are retained so the browser can compute later output without raw reads. Loading performs size, framing, schema, type, uniqueness, sample-reference, conditional consensus/postproc consistency, contamination-mode, collapse-membership, edit-fingerprint, and alignment/tree-reference checks before the UI receives the bundle. Raw reads and spool records are never serialized. A returned Alivibe/manual edit stores its exact nucleotide FASTA, translation frame, baseline/edited/tree fingerprints, warnings, source/time, and optional recalculated Newick tree as a separate copy in the same project file.
The same result object backs the browser explorer, CLI inspect, and all component exports, so exports do not rerun an algorithm or depend on hidden temporary files. “Export all” creates a standard ustar container and gzip-compresses it. The editable .webporpid project and run log are stored at the root; each sample-filtered component is generated through the same exporter as its individual download and placed under a sanitized sample-ID directory.