EOS26

Description

All scripts to run the EOS26 simulation with 21cmFASTv4.2. EOS26 is run phase by phase i.e. ICs first, then PFs, then PHFs, and finally coevals.

How to run

All scripts accept an optional --test flag to run a small test box (HII_DIM=200).

Comparison workflow

• Generate the reference once with --test and no --compare. It is written to EOS26_test_HIIDIM200/.
• Run the production workflow with --compare to check each output against that reference.
• For a local workflow check, use both flags: --test --compare. Candidate files then go to EOS26_test_HIIDIM200_compare/, so the reference is never overwritten. Both test modes use the same input parameters and random seed (42).

End-to-end test

• sbatch sbatch_scripts/full_test_job.sh — runs all three pipeline steps sequentially in test mode.
• qsub pbs_scripts/full_test_job.sh — PBS equivalent of the end-to-end test.

Scaling Measurements

• Run bash scaling/scaling_job.sh to measure the default HII_DIM=200 and HII_DIM=300 cases. Each run disables garbage collection, records peak process RSS with psutil, and amortizes the full node-redshift coeval evolution over its outputs, then writes a phase-by-phase JSON record under scaling/results/. To benchmark other dimensions, pass them as arguments, for example bash scaling/scaling_job.sh 200 300 400.

• After at least two measurements, scaling/run_scalingrelation.py fits power laws for time, peak memory, phase storage, and every stored HDF5 structure. It writes plots, README_scaling_values.md, and scaling_fits.json to scaling/reports/. The generated Markdown file is the source for updating the scaling table below. To apply direct measurements from the smallest and largest completed HII_DIM runs plus fitted extrapolation intervals from every available scaling result, run uv run --no-sync scaling/run_scalingrelation.py --update-readme; it relabels the two measured columns and updates both EOS extrapolation sections.

Production run (in order)

1. Initial conditions (ICs):
   • sbatch sbatch_scripts/ICs_job.sh [--test]

• qsub pbs_scripts/ICs_job.sh [--test] on PBS.
• Writes EOS26.toml (full template with embedded node redshifts and random seed) and EOS26_minimal.toml, then runs run_scripts/run_ICs.py.

2. Perturbed fields (PFs):
   • bash sbatch_scripts/submit_PF_jobs.sh [--test] — submits one job per PF (indices 0–91) (not used for production).
   • sbatch sbatch_scripts/N_PF_job.sh <z_idx> [N] [--test] — runs a batch of N PFs (default N=10) starting from redshift index z_idx.

• bash pbs_scripts/submit_PF_jobs.sh [--test] and qsub pbs_scripts/N_PF_job.sh <z_idx> [N] [--test] are the PBS equivalents.

3. Perturbed halo fields (PHFs):
   • sbatch sbatch_scripts/PHFs_job.sh [--test] — runs run_scripts/run_PHFs.py.

• qsub pbs_scripts/PHFs_job.sh [--test] on PBS.

4. Coevals:
   • sbatch sbatch_scripts/N_coeval_job.sh [N] [--test] — runs a batch of N coevals (default N=10) with run_scripts/run_N_coevals.py.

• qsub pbs_scripts/N_coeval_job.sh [N] [--test] on PBS.

Table

EOS25 simulation step	Computation time [hrs]		Memory [Tb]		Storage [Tb]		SUs
	Estimated	Actual	Estimated	Actual	Estimated	Actual	Estimated	Actual
Initial conditions	13.5 + 2.75 for writing to disk		1.1	1.3	652 Gb	747 Gb		864 EM for ICs + PFs
One perturbed field	0.6		25 Gb	26 Gb	25Gb x 92 = 2.3Tb	2.4Tb
Perturbed halo fields	22 hrs		0.71	0.77	~330 G	33 G	720 EM	617 EM
Evolving astrophysics for one coeval	7		3.65		2.6Tb x 92 = 240 Tb 0.215 x 92 = 20Tb without XRS		672 EM x 92 = 62k

Scaling test results

When updated with --update-readme, each extrapolated value is the mean ± 1σ from all available scaling points. Each extrapolated quantity shows both its current model (affine overhead + coefficient * HII_DIM^3 for peak RSS; free-exponent power law for time and storage) and a fixed-cubic a=3 fit. The regression uncertainty is combined in quadrature with a fixed 10% relative measurement-uncertainty floor. EOS-1: HII_DIM = 1400 (1.5 cMpc/cell, 2100 Mpc). EOS-2: HII_DIM = 1200 (1.667 cMpc/cell, 2000 Mpc). Storage for PFs and coevals is the total across all 92 node redshifts; coeval storage includes retained IonizedBox and excludes transient XraySourceBox.

EOS26 simulation step	Computation time [hrs]		Memory		Storage
	N=100	N=300	N=100	N=300	N=100	N=300
Scaling tests (measured)
Initial conditions	0.0374	0.134	0.939 GB	21.1 GB	0.464 GB	12.5 GB
One perturbed field	0.0345	0.037	0.334 GB	4.49 GB	0.016 GB	0.432 GB
Perturbed halo fields	0.0498	0.17	0.564 GB	11 GB	0.207 GB	5.49 GB
Evolving astrophysics for one coeval	0.00199	0.038	1.16 GB	25 GB	0.0601 GB x 92 = 5.53 GB	1.62 GB x 92 = 149 GB
Extrapolated to EOS-1 (HII_DIM = 1400, 1.5 cMpc/cell, 2100 Mpc)
Initial conditions	a=1.13: 0.701 ± 0.307	a=3: 14.5 ± 3.28	affine: 2.13 ± 0.213 TB	a=3: 2.15 ± 0.216 TB	a=3: 1.27 ± 0.122 TB	a=3: 1.28 ± 0.128 TB
One perturbed field	a=0.06: 0.0402 ± 0.00434	a=3: 4.54 ± 2.81	affine: 0.439 ± 0.0439 TB	a=3: 0.46 ± 0.0481 TB	a=3: 0.0438 ± 0.00418 TB	a=3: 0.0439 ± 0.00439 TB
Perturbed halo fields	a=1.06: 0.753 ± 0.644	a=3: 18.1 ± 3.86	affine: 1.1 ± 0.11 TB	a=3: 1.13 ± 0.114 TB	a=2.98: 0.542 ± 0.0523 TB	a=3: 0.557 ± 0.0557 TB
Evolving astrophysics for one coeval	a=2.69: 2.43 ± 0.295	a=3: 3.92 ± 0.415	affine: 2.51 ± 0.251 TB	a=3: 2.54 ± 0.255 TB	a=3: 0.164 ± 0.0122 TB x 92 = 15.1 ± 1.12 TB	a=3: 0.165 ± 0.0128 TB x 92 = 15.1 ± 1.18 TB
Extrapolated to EOS-2 (HII_DIM = 1200, 1.667 cMpc/cell, 2000 Mpc)
Initial conditions	a=1.13: 0.589 ± 0.24	a=3: 9.13 ± 2.07	affine: 1.34 ± 0.134 TB	a=3: 1.35 ± 0.135 TB	a=3: 0.802 ± 0.0765 TB	a=3: 0.804 ± 0.0804 TB
One perturbed field	a=0.06: 0.0398 ± 0.00423	a=3: 2.86 ± 1.77	affine: 0.277 ± 0.0277 TB	a=3: 0.29 ± 0.0303 TB	a=3: 0.0276 ± 0.00263 TB	a=3: 0.0276 ± 0.00276 TB
Perturbed halo fields	a=1.06: 0.639 ± 0.501	a=3: 11.4 ± 2.43	affine: 0.695 ± 0.0696 TB	a=3: 0.711 ± 0.0715 TB	a=2.98: 0.342 ± 0.033 TB	a=3: 0.351 ± 0.0351 TB
Evolving astrophysics for one coeval	a=2.69: 1.61 ± 0.19	a=3: 2.47 ± 0.262	affine: 1.58 ± 0.158 TB	a=3: 1.6 ± 0.161 TB	a=3: 0.103 ± 0.0077 TB x 92 = 9.51 ± 0.708 TB	a=3: 0.104 ± 0.00809 TB x 92 = 9.54 ± 0.744 TB

Projected maximum simulation size

Evolving astrophysics for one coeval is the most memory-intensive phase at every measured HII_DIM (see table above), so it sets the ceiling for the largest simulation a single node can run. Peak memory for 3D simulation grids is physically fixed per-process overhead (interpreter, shared libraries, lookup tables) plus a term that scales with box volume (HII_DIM^3), so peak RSS is fit as overhead + coefficient * HII_DIM^3 by ordinary least squares using all measured points (HII_DIM = 100, 200, 300), rather than a free-exponent log-log power law, which is biased low by that same fixed overhead when it is not negligible at small HII_DIM (the same effect documented above for compute time). Resolution is fixed at 1.5 cMpc/cell, matching all current scaling points; only HII_DIM is varied. The "Max HII_DIM" columns require the upper end of the fit's 1-sigma band to stay within the memory budget, so the true peak should stay at or below the stated value.

Memory budget	Max HII_DIM	Box length [Mpc]	Predicted peak RSS (coeval)
3.0 TB	1439	2158	2.72 ± 0.273 TB
2.7 TB (90% safety margin)	1390	2085	2.45 ± 0.246 TB

Predicted coeval peak RSS at the current EOS target sizes (affine overhead + coefficient * HII_DIM^3 fit on all measured points):

EOS target	HII_DIM	Predicted peak RSS (coeval)
EOS-1	1400	2.51 ± 0.251 TB
EOS-2	1200	1.58 ± 0.158 TB

With only 3 measured points, the affine fit above is pulled noticeably off the data by whichever point dominates the linear-space sum of squares. A free-exponent power-law fit (the same form used for time and storage; see fit_power_law) tracks the coeval measurements markedly better, so it is shown here for comparison -- the affine model above remains the one used for the Max HII_DIM budget table:

EOS target	HII_DIM	Predicted peak RSS, affine	Predicted peak RSS, power law
EOS-1	1400	2.51 ± 0.251 TB	1.8 ± 0.225 TB
EOS-2	1200	1.58 ± 0.158 TB	1.17 ± 0.142 TB

Caveats: this projection extrapolates well beyond the largest measured HII_DIM (300) to production scale, so treat it as an order-of-magnitude estimate rather than a precise bound. Run an additional scaling point at HII_DIM = 400 (or larger) on the cluster and check whether the residuals from the affine fit stay small -- growing residuals at larger HII_DIM would mean memory grows faster than the assumed overhead-plus-cubic form and this projection is optimistic. Consider also profiling one run with memray to see the full memory-vs-redshift curve and confirm the RSS sampler is not missing any brief spikes.

EOS-1 Fixed-Cubic Runtime Plan

This planning table uses the fixed a=3 central values at HII_DIM=1400. It assumes a 1.5 h read of the ICs for every job and uses a documented 3 h full-IC write to estimate output-write time. Coeval output excludes the 4D XraySourceBox. Four coevals per job is deliberately conservative under Gadi's 24-hour maximum walltime when all 96 cores or ~3TB mem are requested. Peak RSS is the per-phase process estimate. The serial column is total work if batches run one after another. The PFs will actually be run in parallel though.

Phase	Work unit	Units/job	Compute/unit [h]	IC read/job [h]	Write/unit [h]	Estimated job walltime [h]	Jobs for full phase	Serial phase walltime [h]	Peak RSS [TB]	Output, full phase [TB]
Initial conditions	all ICs	1	14.50	0.00	3.00	17.50	1	17.5	2.15	1.28
Perturbed fields	one PF	4	4.54	1.50	0.10	20.08	23	461.8	0.46	4.04
Perturbed halo fields	all halo fields	1	18.12	1.50	1.31	20.93	1	20.9	1.13	0.56
Coevals	one coeval	4	3.92	1.50	0.39	18.75	23	431.2	2.54	15.1