Tempo version 3 by AndersJensen-NOAA · Pull Request #3078 · ufs-community/ufs-weather-model

AndersJensen-NOAA · 2026-02-04T18:44:32Z

Commit Queue Requirements:

This PR addresses a relevant WM issue (if not, create an issue).
All subcomponent pull requests (if any) have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines), preferably on Ursa (Derecho or Hercules are acceptable alternatives). Exceptions: documentation-only PRs, CI-only PRs, etc.
- Commit log file w/full results from RT suite run (if applicable).
- Verify that test_changes.list indicates which tests, if any, are changed by this PR. Commit test_changes.list, even if it is empty.
Fill out all sections of this template.

Description:

Updates to tempo microphysics version 3: Addresses issue: #3109

Commit Message:

* UFSWM - 
  * UFSATM - 
    * ccpp-physics -

Priority:

Critical Bugfix: Reason
High: Reason
Normal

Git Tracking

UFSWM:

Closes #3109

Sub component Pull Requests:

UFSATM: Tempo version 3 NOAA-EMC/ufsatm#1063
- CCPP: Tempo version 3 ccpp-physics#353

UFSWM Blocking Dependencies:

Blocked by #
None

Documentation:

Documentation update required.
- Relevant updates are included with this PR.
- A WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
Documentation update NOT required.
- Explanation: tempo has its own documentation

Changes

Regression Test Changes (Please commit test_changes.list):

PR Adds New Tests/Baselines.
PR Updates/Changes Baselines.
No Baseline Changes.

Input data Changes:

None.
PR adds input data.
PR changes existing input data. New lookup tables are needed

Library Changes/Upgrades:

Required
- Library names w/versions:
- Git Stack Issue (JCSDA/spack-stack#)
No Updates

Testing Log:

gspetro-NOAA · 2026-02-23T16:29:57Z

@AndersJensen-NOAA Could you please fill out the PR template if this PR is ready? We can't move the PR forward until we have the info requested in the template. If the PR is not ready, it would be best to convert it to draft form until it is ready.

AndersJensen-NOAA · 2026-06-15T17:41:59Z

@grantfirl I'm happy to pull down code and fire off tests tonight.

AndersJensen-NOAA · 2026-06-16T14:58:17Z

@grantfirl new tests are here: /scratch4/BMC/wrfruc/jensen/ufs_tempo310/tests

AndersJensen-NOAA · 2026-06-16T15:02:43Z

@grantfirl I also made a new test regional_wofs_tempo as part of this PR (replacing the p8 hail test), but I don't think that it ran.

grantfirl · 2026-06-16T15:52:22Z

scratch4/BMC/wrfruc/jensen/ufs_tempo310/tests

@AndersJensen-NOAA Thanks for running those. I'll push the RT log. I think that regional_wofs_tempo test didn't run because it wasn't added to rt.conf. I'll add it and try to run just that test on Ursa and upload the log in the comments to prove that it runs.

grantfirl · 2026-06-16T16:49:01Z

@AndersJensen-NOAA The regional_wofs_intel test now runs with the changes in 82dcaec

RegressionTests_ursa.log

grantfirl · 2026-06-16T16:55:11Z

@gspetro-NOAA The control_p8_ugwpv1_tempo_aerosol_hail test was removed from Ursa due to memory issues a few PRs back. It might be able to be turned back on with the addition of memory for this test by @AndersJensen-NOAA in this PR. I'll leave that up to you all to decide.

gspetro-NOAA · 2026-06-16T16:57:53Z

@AndersJensen-NOAA @grantfirl I have copied the input-data-20251015 directory contents to input-data-20260617. From input-data-20260617, I plan to remove:

FV3_input_data_TEMPO/MP_TEMPO_freezeH2O.dat
FV3_input_data_TEMPO/MP_TEMPO_QRacrQG.dat
FV3_input_data_TEMPO/MP_TEMPO_HAILAWARE_QRacrQG.dat
FV3_input_data_TEMPO/MP_TEMPO_QRacrQS.dat
FV3_fix/CCN_ACTIVATE.BIN

I need to add:

/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qs_data_tempo_v3
/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qg_data_tempo_v3
/scratch4/BMC/wrfruc/jensen/for_ftp/freeze_water_data_tempo_v3
to input-data-20260617/FV3_input_data_TEMPO/.

I'll also need to add /scratch4/BMC/wrfruc/jensen/for_ftp/ccn_activate.bin to input-data-20260617/FV3_fix.

Is that correct?

grantfirl · 2026-06-16T16:57:55Z

@gspetro-NOAA I added the testing logs from the runs that completed last Friday, although there were some more minor changes in the TEMPO submodule since then. I don't anticipate any of these changes causing any trouble, but I can't prove that without rerunning the pre-tests. Please let me know if you'd like them to be re-run before testing on other platforms.

Also, it looks like that there are more compiler warnings. I'll see if I can track those down.

The additional compiler warnings disappeared after updating the code to the latest develop branches. I'm not sure what the extra remarks are about.

grantfirl · 2026-06-16T17:02:11Z

@AndersJensen-NOAA @grantfirl I have copied the input-data-20251015 directory contents to input-data-20260617. From input-data-20260617, I plan to remove:

FV3_input_data_TEMPO/MP_TEMPO_freezeH2O.dat

FV3_input_data_TEMPO/MP_TEMPO_QRacrQG.dat

FV3_input_data_TEMPO/MP_TEMPO_HAILAWARE_QRacrQG.dat

FV3_input_data_TEMPO/MP_TEMPO_QRacrQS.dat

FV3_fix/CCN_ACTIVATE.BIN

I need to add:

/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qs_data_tempo_v3

/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qg_data_tempo_v3

/scratch4/BMC/wrfruc/jensen/for_ftp/freeze_water_data_tempo_v3
to input-data-20260617/FV3_input_data_TEMPO/.

I'll also need to add /scratch4/BMC/wrfruc/jensen/for_ftp/ccn_activate.bin to input-data-20260617/FV3_fix.

Is that correct?

In addition to updating the *_run.IN files afterward to reflect the newly-staged files, I think that this is correct, although I'll let @AndersJensen-NOAA confirm.

gspetro-NOAA · 2026-06-16T17:02:23Z

@gspetro-NOAA The control_p8_ugwpv1_tempo_aerosol_hail test was removed from Ursa due to memory issues a few PRs back. It might be able to be turned back on with the addition of memory for this test by @AndersJensen-NOAA in this PR. I'll leave that up to you all to decide.

I think it would be good to turn it back on and test whether it works.

NickSzapiro-NOAA · 2026-06-16T17:16:23Z


  export TPN=192
-
+  export MEM_PER_NODE="368640M"


where does this default MEM_PER_NODE
(used in Ursa #SBATCH --mem=@[MEM_PER_NODE])
come from?

Isn't this more than the 300G set in regional_wofs_tempo‎?

By default, Ursa gives 1.92G per core, and there are 192 cores/node. For a default TPN=192, the default memory should be 368640M. This memory amount is a system-level setting on Ursa. Note that Ursa has 2G per core available.

The reason why the tempo_p8 tests need 300G of memory is because they use 150 tasks, and by default, this does not provide enough memory for tempo.
see this comment: ufs-community/ccpp-physics#353 (comment)

If the regional_wofs_tempo test uses TPN=192, then it doesn't need the export MEM_PER_NODE="300G" statement.

I was kinda confused by the memory changes for Ursa too. With the addition of export MEM_PER_NODE="368640M" in default_vars.sh, this is setting the memory for every Ursa test, right? Why is this needed if this is the same as the system-level setting on Ursa? Then, the individual TEMPO tests are asking for export MEM_PER_NODE="300G", which is now less than the default. This is what is desired? Or do I have this wrong?

I added the export MEM_PER_NODE="300G" statement to regional_wofs_tempo. I thought it was necessary. Where are the number of tasks being set for the tempo_p8 tests that is different than the default?

Thanks for the details, explanation, and sorting through the intermittent test failures

Does this fix for memory in one scheme practically change quite a few unrelated tests that also have TPN/=192?

If so, one option would be to comment this MEM_PER_NODE setting by default, similar to EXCLUSIVE_NODES_OPT or HIDE_UGWPV1 ideas

It would be more work to compute these based on system settings near compute_petbounds_and_tasks

Yes other tests with TPN/=192 would be changed. I don't know how to comment, or hide, a variable in the default settings though. But that would be the best solution.

One way is like AndersJensen-NOAA#3

AndersJensen-NOAA · 2026-06-16T17:38:38Z

@AndersJensen-NOAA @grantfirl I have copied the input-data-20251015 directory contents to input-data-20260617. From input-data-20260617, I plan to remove:

FV3_input_data_TEMPO/MP_TEMPO_freezeH2O.dat

FV3_input_data_TEMPO/MP_TEMPO_QRacrQG.dat

FV3_input_data_TEMPO/MP_TEMPO_HAILAWARE_QRacrQG.dat

FV3_input_data_TEMPO/MP_TEMPO_QRacrQS.dat

FV3_fix/CCN_ACTIVATE.BIN

I need to add:

/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qs_data_tempo_v3

/scratch4/BMC/wrfruc/jensen/for_ftp/qr_acr_qg_data_tempo_v3

/scratch4/BMC/wrfruc/jensen/for_ftp/freeze_water_data_tempo_v3
to input-data-20260617/FV3_input_data_TEMPO/.

I'll also need to add /scratch4/BMC/wrfruc/jensen/for_ftp/ccn_activate.bin to input-data-20260617/FV3_fix.

Is that correct?

Yes, this is correct. Delete the old tables, and add the new ones. The ccn_activate.bin table should be the same one that Thompson uses. The other tables are different.

gspetro-NOAA · 2026-06-17T18:04:52Z

@AndersJensen-NOAA Could you accept (or reject, if problematic) my proposed changes so that I can test the data staging for the PR?

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

AndersJensen-NOAA · 2026-06-17T18:28:45Z

@AndersJensen-NOAA Could you accept (or reject, if problematic) my proposed changes so that I can test the data staging for the PR?

@gspetro-NOAA You should be good to test now.

Replace MEM_PER_NODE with MEM_PER_NODE_OPT

AndersJensen-NOAA added 2 commits February 4, 2026 18:23

tempo version 3

b390aa4

tempo version 3

69492cb

github-project-automation Bot added this to PRs to Process Feb 4, 2026

github-project-automation Bot moved this to Evaluating in PRs to Process Feb 4, 2026

AndersJensen-NOAA marked this pull request as draft February 4, 2026 18:44

AndersJensen-NOAA added 7 commits February 13, 2026 14:27

update pointer to UFSATM

71ae592

update pointer to ufsatm (fv3atm)

2e63aed

update point to ufsatm

2722b67

update pointer

0178e4a

update pointer to ufsatm

adea8b4

update pointer to fv3

91b4958

update pointer to ufsatm

7eaceca

AndersJensen-NOAA marked this pull request as ready for review February 19, 2026 22:13

AndersJensen-NOAA added 2 commits February 20, 2026 19:38

update pointer to ufsatm

071c088

ufsatm pointer

71d9ad2

gspetro-NOAA moved this from Evaluating to Not Ready in PRs to Process Feb 25, 2026

update to tempo version 3.0.4

1eb5ba4

This was referenced Mar 9, 2026

Tempo version 3 ufs-community/ccpp-physics#353

Open

Tempo version 3 NOAA-EMC/ufsatm#1063

Open

AndersJensen-NOAA added 2 commits March 17, 2026 18:27

updates to CCPP from code review

881211d

update tempo

12b5283

gspetro-NOAA mentioned this pull request May 1, 2026

control_p8_ugwpv1_tempo_aerosol_hail frequently has OOM failures on Ursa #3094

Open

AndersJensen-NOAA added 7 commits May 13, 2026 19:23

update pointer

3574fc0

update pointer

bd1803c

Merge branch 'develop' into tempo_v3

766a90f

Add additional memory for Tempo on ursa to prevent OOM issues

27b5750

RTs

eeb7d1c

update pointer

acf88ec

update

b94e760

grantfirl added 2 commits June 16, 2026 15:53

add new RT log from Anders

ba6fb8d

add regional_wofs_tempo to rt.conf

82dcaec

gspetro-NOAA moved this from Pre-testing required to Waiting for Reviews (subcomponent) in PRs to Process Jun 16, 2026

grantfirl added 2 commits June 16, 2026 12:50

update test_changes.list to include regional_wofs_tempo intel

00122d5

Merge branch 'develop' into tempo_v3

adfcfee

NickSzapiro-NOAA reviewed Jun 16, 2026

View reviewed changes

gspetro-NOAA reviewed Jun 16, 2026

View reviewed changes

Comment thread tests/fv3_conf/control_run.IN Outdated

gspetro-NOAA reviewed Jun 16, 2026

View reviewed changes

Comment thread tests/fv3_conf/control_run.IN Outdated

gspetro-NOAA reviewed Jun 16, 2026

View reviewed changes

Comment thread tests/fv3_conf/regional_run.IN Outdated

gspetro-NOAA reviewed Jun 16, 2026

View reviewed changes

Comment thread tests/fv3_conf/regional_run.IN Outdated

Update INPUTDATA_ROOT to input-data-20260617

8429960

AndersJensen-NOAA and others added 4 commits June 17, 2026 12:27

Update tests/fv3_conf/control_run.IN

26c4ca4

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

Update tests/fv3_conf/control_run.IN

790aba6

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

Update tests/fv3_conf/regional_run.IN

ca13ca4

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

Update tests/fv3_conf/regional_run.IN

372e88c

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

NickSzapiro-NOAA and others added 2 commits June 17, 2026 19:10

Replace MEM_PER_NODE with MEM_PER_NODE_OPT

1fd8a52

Merge pull request #3 from NickSzapiro-NOAA/tempo_v3

26823c3

Replace MEM_PER_NODE with MEM_PER_NODE_OPT

Conversation

AndersJensen-NOAA commented Feb 4, 2026 • edited by gspetro-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Documentation:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

Uh oh!

gspetro-NOAA commented Feb 23, 2026

Uh oh!

AndersJensen-NOAA commented Jun 15, 2026

Uh oh!

AndersJensen-NOAA commented Jun 16, 2026

Uh oh!

AndersJensen-NOAA commented Jun 16, 2026

Uh oh!

grantfirl commented Jun 16, 2026

Uh oh!

grantfirl commented Jun 16, 2026

Uh oh!

grantfirl commented Jun 16, 2026

Uh oh!

gspetro-NOAA commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grantfirl commented Jun 16, 2026

Uh oh!

grantfirl commented Jun 16, 2026

Uh oh!

gspetro-NOAA commented Jun 16, 2026

Uh oh!

NickSzapiro-NOAA Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

AndersJensen-NOAA Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

grantfirl Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

NickSzapiro-NOAA Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

AndersJensen-NOAA Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

NickSzapiro-NOAA Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndersJensen-NOAA commented Jun 16, 2026

Uh oh!

gspetro-NOAA commented Jun 17, 2026

Uh oh!

AndersJensen-NOAA commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndersJensen-NOAA commented Feb 4, 2026 •

edited by gspetro-NOAA

Loading

gspetro-NOAA commented Jun 16, 2026 •

edited

Loading