Repository navigation
Commit 7b9c81e
authored
Selene manager: store work proportional to changes, not record count (#139)
* Selene manager: build the metrics text when it is scraped
The owner thread rebuilt the whole metrics text after every command batch, walking every record twice per namespace. A scrape now asks the owner for the text, so an idle store does no per-record work and a busy one does it once per scrape.
Design-Signed-Off-By: yatesdr; design-review=1c01
* Selene manager: refresh durability only for records that can change
Every publish, disk-write completion and compaction completion recomputed durability for every record in the namespace, sorted by depth. The store now keeps the set of records that are not durable yet and a refresh visits the touched record, and that set only when the record has children and has just become durable; replacing a durable record still rescans, since its descendants may stop being durable. Recovery still rebuilds the set from the whole tree.
Design-Signed-Off-By: yatesdr; design-review=1c01
* Selene manager: schedule disk writes from the not-yet-durable set
schedule() walked every record in every namespace on each pump to find the few that still need a disk write. It now walks the set of records that are not durable yet, which the durability refresh already maintains.
Design-Signed-Off-By: yatesdr; design-review=1c01
* Selene manager: build the catalog snapshot without sorting
The snapshot sorted every key before building the entries, but its only consumer upserts them into a hash index, so order is unspecified and the sort is gone.
Design-Signed-Off-By: yatesdr; design-review=1c01
* Selene manager: iterate a bucket in place instead of copying and sorting it
in_bucket cloned and sorted the whole bucket on every boundary of every lookup. The bucket is now an ordered set, so the lookup iterates it without a copy, in value order, which is the order the old sort produced; the candidate order and the scan budget cutoff are unchanged.
Design-Signed-Off-By: yatesdr; design-review=1c01
* Selene manager: scale test for a publish into a large store
An ignored, release-only test that holds 100k records and a 10k backlog of records not yet on disk, asserts the work after a publish (durability refresh and scheduling) takes under a millisecond, and checks that the incremental refreshes agree with a full rescan.
Design-Signed-Off-By: yatesdr; design-review=1c011 parent 9920d90 commit 7b9c81e
4 files changed
Lines changed: 271 additions & 50 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
235 | 235 | | |
236 | 236 | | |
237 | 237 | | |
238 | | - | |
| 238 | + | |
| 239 | + | |
239 | 240 | | |
240 | 241 | | |
241 | 242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
184 | 183 | | |
185 | 184 | | |
186 | 185 | | |
| |||
211 | 210 | | |
212 | 211 | | |
213 | 212 | | |
214 | | - | |
| 213 | + | |
215 | 214 | | |
216 | 215 | | |
217 | 216 | | |
218 | 217 | | |
219 | | - | |
220 | 218 | | |
221 | 219 | | |
222 | 220 | | |
| |||
253 | 251 | | |
254 | 252 | | |
255 | 253 | | |
256 | | - | |
257 | 254 | | |
258 | 255 | | |
259 | 256 | | |
| |||
454 | 451 | | |
455 | 452 | | |
456 | 453 | | |
457 | | - | |
| 454 | + | |
458 | 455 | | |
459 | 456 | | |
460 | 457 | | |
461 | 458 | | |
| 459 | + | |
462 | 460 | | |
463 | 461 | | |
464 | 462 | | |
| |||
468 | 466 | | |
469 | 467 | | |
470 | 468 | | |
471 | | - | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
472 | 480 | | |
473 | 481 | | |
474 | 482 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
| 122 | + | |
| 123 | + | |
130 | 124 | | |
131 | 125 | | |
132 | 126 | | |
| |||
257 | 251 | | |
258 | 252 | | |
259 | 253 | | |
260 | | - | |
| 254 | + | |
261 | 255 | | |
262 | 256 | | |
263 | 257 | | |
| |||
279 | 273 | | |
280 | 274 | | |
281 | 275 | | |
282 | | - | |
| 276 | + | |
283 | 277 | | |
284 | 278 | | |
285 | 279 | | |
| |||
0 commit comments