[RL] - MessageEnv, Rollout types, Rubric, Renderer - #3453

Merged

felipemello1 merged 40 commits into

pytorch:mainfrom

felipemello1:v8-datatypes-env-protocol

Jun 5, 2026

Commits on May 20, 2026

[RL] Add Batcher as Configurable, support [B, L] microbatches
wwwjn
committed
6913120
rebase
wwwjn
committed
7c7c9f7
remove position in loss calc
wwwjn
committed
8e4fc60
update algorithm
wwwjn
committed
dc690a1
update rebase
wwwjn
committed
40ef4c5
update trainer to using new TrainBatch
wwwjn
committed
767022f
update loss
wwwjn
committed
50b5572
update names and configs
wwwjn
committed
86ed252
update batcher with waste metrics
wwwjn
committed
31dcd8b

Commits on May 22, 2026

update configs
wwwjn
committed
7dad6f5

Commits on May 26, 2026

address comments
wwwjn
committed
d440e46

Commits on May 27, 2026

v8 datatypes + env protocol
Felipe Mello
committed
1523db5
v8 rubric + rollouts refactor
Felipe Mello
committed
312c292

Commits on May 28, 2026

v8 Task surface refactor + style pass
Felipe Mello
committed
14aaeca

Commits on May 29, 2026

v8 docstring/style pass + truncation fix + renderer tokenizer
Felipe Mello
committed
6292f1d
Merge upstream/main (landed batcher PR) into v8 branch
Felipe Mello
committed
e9af27a
Fix NoReduce collision: rollout/group_failures sums across collection rounds
Felipe Mello
committed
85880db
Adopt renderers typed-config API (create_renderer(tokenizer, config))
Felipe Mello
committed
9243ba1
Revert unrelated merge drift to upstream/main
Felipe Mello
committed
701b4d1
RL rollout loop cleanup: RolloutGroup, per-rollout step, status enum
Felipe Mello
committed
11003ba
RL example default: thinking on, max_tokens=700; tidy rollout types
Felipe Mello
committed
5044e86
RL env/types review pass (docs 69+70): renames, env reward, renderer build
Felipe Mello
committed
2dcd0f9
Fix _build_episodes group-skip + rollout_to_episode text
Felipe Mello
committed
7a72a41
Docstring concision pass + rename envs/ -> env_types/
Felipe Mello
committed
0cf8460

Commits on May 30, 2026

Make Task an ABC; base Config owns rubric, subclass builds it
Felipe Mello
committed
df376bb

Commits on Jun 2, 2026

[rl] v8: message/token env types, Configurable rubric, Task ABC
Felipe Mello
committed
d3ba2df

Commits on Jun 3, 2026

Refine RL rollout protocol
Felipe Mello
committed
de7c887
Rename Task.Config.env_config -> env_wrapper_cfg
Felipe Mello
committed
c2efd2b
Merge remote-tracking branch 'upstream/main' into v8-datatypes-env-protocol
Felipe Mello
committed
8b597ba
[rl] address PR review (round 2): env/rollout naming + cleanups
Felipe Mello
committed
79c3903

Commits on Jun 4, 2026

[rl] PR review round 2: shape hints + Task->Rollouter rename
Felipe Mello
committed
621e3cf
[rl] PR review round 2: env/rollout/examples package layout
Felipe Mello
committed
3c69ae6
[rl] renderer: None-default knobs; build the typed config via create_renderer
Felipe Mello
committed
c77d47f
Merge remote-tracking branch 'upstream/main' into v8-datatypes-env-protocol
Felipe Mello
committed
75e69d4

Commits on Jun 5, 2026

[rl] PR review round 3: naming (completion_message, MessageEnv*Output, sample), docstrings + TODOs
Felipe Mello
committed
80104f5
[rl] PR review round 3: get_train_sample -> get_training_sample (tianyu nit)
Felipe Mello
committed
4bf9b33
Merge branch 'main' of https://github.com/pytorch/torchtitan into v8-datatypes-env-protocol
Felipe Mello
committed
9d91d4c
[rl] token: simplify the history-edit TODO comment
Felipe Mello
committed
d33903e
[rl] rollout_to_episode: TODO for branching when multi-turn history diverges
Felipe Mello
committed
89f8f85
[rl] CI: install renderers in RL integration test workflows
Felipe Mello
committed
04c47c2