forked from science-of-finetuning/diffing-toolkit
-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathtodo
More file actions
30 lines (20 loc) · 1.5 KB
/
Copy pathtodo
File metadata and controls
30 lines (20 loc) · 1.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# testing
- logit diff topk tests
- agent scores with ADL at different layers
## exp
why fda has an adl so low on fda and 1:1 has a datpoint whiele there is no modek?
- make sure that the exps overwrite instead of adding results to avoid illusion of smaller variance.
# amplification dashboard
- multi prompt add continue regenerate etc
- results in multi prompt should have the same widgets as in the multi gen tab.
- cut the debugging or at least put in if debug mode is enabled.
- fix the fact that we need to reload the page once to load the multi gen tab cache
## refactoring?
- flatten `method_params` nesting in `kl` and `activation_analysis` configs/code to match `diff_mining`/`adl` style (params at top level, no `method_params` key)
- check for duplicated code between multi gen and multi prompt prompt building functions
- split the dashboard into subclasses (which store a reference to the main dashboard)
# clement
- none should not be a config (clement 1 day later: ???) (clement 1 week later: yeah the fact that is a config is a bit hacky, maybe there is some better way to do this)
- fix _MODEL_CACHE being dumb when using dashboard, we should probably use the streamlit cache instead?
- reduce code duplication in diffing methods (e.g. pca, sae_difference, activation_analysis all have the same code for computing activations in both models from the same input). See docs/DUPLICATION_REPORT.md for more details.
- support full finetunes in ActivationOracleMethod (currently only LoRA adapters supported)