MPE preprint version#215

Open

FarmersWrap wants to merge 61 commits into

texttron:mainfrom

FarmersWrap:preprint

Commits on Dec 10, 2025

rewrote passage chunking
Ryan Yu
committed
added logic for left padding
Ryan Yu
committed

Commits on Dec 11, 2025

added search
Ryan Yu
committed

Commits on Dec 14, 2025

changed the tokenizer logic
Ryan Yu
committed
added train collator debug
Ryan Yu
committed
traincollator is done
Ryan Yu
committed
fixed some comments
Ryan Yu
committed
modified chunkedencoder
Ryan Yu
committed
Modified forward and model
Ryan Yu
committed
Modified inference on chunked passage, in progress
Ryan Yu
committed
fixed a chunk size not passed to model
Ryan Yu
committed

Commits on Dec 15, 2025

changed eos to sep
Ryan Yu
committed

Commits on Dec 16, 2025

added logs
Ryan Yu
committed

Commits on Dec 19, 2025

added some scripts
Ryan Yu
committed

Commits on Dec 22, 2025

added tests
Ryan Yu
committed
added tests
Ryan Yu
committed
added log
Ryan Yu
committed

Commits on Dec 23, 2025

review
Ryan Yu
committed
added collator helper functions
Ryan Yu
committed
added padding helper
Ryan Yu
committed
Tested the collator
Ryan Yu
committed
Reviewed forward and maxsim
Ryan Yu
committed

Commits on Dec 24, 2025

dataset uses random negative for cases have less negatives
Ryan Yu
committed
added some prints
Ryan Yu
committed
removed one breakpoint
Ryan Yu
committed

Commits on Dec 29, 2025

Added random chunks
Ryan Yu
committed
Added full randomization
Ryan Yu
committed
removed useless variables
Ryan Yu
committed
Refactored the randomization
Ryan Yu
committed

Commits on Jan 6, 2026

added search time passage encoding with chunks
Ryan Yu
committed

Commits on Jan 10, 2026

Added random chunk size for eval, and tests for prechunked passages and random chunking
Ryan Yu
committed

Commits on Jan 26, 2026

Move loss compute to trainer
Ryan
committed
add logs
Ryan Yu
committed
try to fix masks
Ryan Yu
committed
removed some logs
kevin-zkc
committed
Removed many logs
Ryan
committed
removed emb printing
Ryan
committed

Commits on Jan 27, 2026

one more fix for chunk size
Ryan
committed
add debug log
Ryan Yu
committed
fix hash
Ryan Yu
committed

Commits on Jan 29, 2026

added script generator
Ryan Yu
committed
updated the scripts
Ryan Yu
committed
modified script
Ryan Yu
committed
changed the scripts and log
Ryan Yu
committed
report to wandb
Ryan Yu
committed

Commits on Jan 30, 2026

reevaluated the code
Ryan Yu
committed

Commits on Feb 6, 2026

Commits on Feb 13, 2026

Added diff
Ryan Yu
committed
added query
Ryan Yu
committed
added tests/test_chunking.py
Ryan Yu
committed

Commits on Feb 27, 2026

Added mpe
Ryan Yu
committed
deleted some scripts
Ryan Yu
committed
deleted some scripts-2
Ryan Yu
committed
deleted non-necessary code
Ryan Yu
committed
deleted more scripts
Ryan Yu
committed
dataset package
Ryan Yu
committed
added req
Ryan Yu
committed
uses sdpa for longembed
Ryan Yu
committed

Commits on Feb 28, 2026

changed readme
Ryan Yu
committed