-
Notifications
You must be signed in to change notification settings - Fork 279
Tempo version 3 #3078
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AndersJensen-NOAA
wants to merge
38
commits into
ufs-community:develop
Choose a base branch
from
AndersJensen-NOAA:tempo_v3
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Tempo version 3 #3078
Changes from 31 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
b390aa4
tempo version 3
AndersJensen-NOAA 69492cb
tempo version 3
AndersJensen-NOAA 71ae592
update pointer to UFSATM
AndersJensen-NOAA 2e63aed
update pointer to ufsatm (fv3atm)
AndersJensen-NOAA 2722b67
update point to ufsatm
AndersJensen-NOAA 0178e4a
update pointer
AndersJensen-NOAA adea8b4
update pointer to ufsatm
AndersJensen-NOAA 91b4958
update pointer to fv3
AndersJensen-NOAA 7eaceca
update pointer to ufsatm
AndersJensen-NOAA 071c088
update pointer to ufsatm
AndersJensen-NOAA 71d9ad2
ufsatm pointer
AndersJensen-NOAA 1eb5ba4
update to tempo version 3.0.4
AndersJensen-NOAA 881211d
updates to CCPP from code review
AndersJensen-NOAA 12b5283
update tempo
AndersJensen-NOAA 3574fc0
update pointer
AndersJensen-NOAA bd1803c
update pointer
AndersJensen-NOAA 766a90f
Merge branch 'develop' into tempo_v3
AndersJensen-NOAA 27b5750
Add additional memory for Tempo on ursa to prevent OOM issues
AndersJensen-NOAA eeb7d1c
RTs
AndersJensen-NOAA acf88ec
update pointer
AndersJensen-NOAA b94e760
update
AndersJensen-NOAA ac9a49a
Merge branch 'develop' into tempo_v3
grantfirl 659b9c7
update tempo
AndersJensen-NOAA e84d9e0
Merge branch 'tempo_v3' of https://github.com/AndersJensen-NOAA/ufs-w…
AndersJensen-NOAA 16bdbb1
update tempo
AndersJensen-NOAA a2ee37c
Merge branch 'develop' into tempo_v3
grantfirl 90ce4db
add Ursa logs
grantfirl ba6fb8d
add new RT log from Anders
grantfirl 82dcaec
add regional_wofs_tempo to rt.conf
grantfirl 00122d5
update test_changes.list to include regional_wofs_tempo intel
grantfirl adfcfee
Merge branch 'develop' into tempo_v3
grantfirl 8429960
Update INPUTDATA_ROOT to `input-data-20260617`
gspetro-NOAA 26c4ca4
Update tests/fv3_conf/control_run.IN
AndersJensen-NOAA 790aba6
Update tests/fv3_conf/control_run.IN
AndersJensen-NOAA ca13ca4
Update tests/fv3_conf/regional_run.IN
AndersJensen-NOAA 372e88c
Update tests/fv3_conf/regional_run.IN
AndersJensen-NOAA 1fd8a52
Replace MEM_PER_NODE with MEM_PER_NODE_OPT
NickSzapiro-NOAA 26823c3
Merge pull request #3 from NickSzapiro-NOAA/tempo_v3
AndersJensen-NOAA File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule UFSATM
updated
9 files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,144 +1,3 @@ | ||
| cpld_control_gfsv17 intel | ||
| cpld_control_gfsv17_iau intel | ||
| cpld_restart_gfsv17 intel | ||
| cpld_restart_gfsv17_iau intel | ||
| cpld_mpi_gfsv17 intel | ||
| cpld_control_p8 intel | ||
| cpld_control_p8.v2.sfc intel | ||
| cpld_restart_p8 intel | ||
| cpld_2threads_p8 intel | ||
| cpld_decomp_p8 intel | ||
| cpld_mpi_p8 intel | ||
| cpld_control_ciceC_p8 intel | ||
| cpld_control_noaero_p8_agrid intel | ||
| cpld_control_c48_5deg intel | ||
| cpld_warmstart_c48_5deg intel | ||
| cpld_restart_c48_5deg intel | ||
| cpld_control_c24_5deg intel | ||
| cpld_warmstart_c24_5deg intel | ||
| cpld_restart_c24_5deg intel | ||
| cpld_control_c24_9deg intel | ||
| cpld_warmstart_c24_9deg intel | ||
| cpld_restart_c24_9deg intel | ||
| cpld_control_c12_9deg intel | ||
| cpld_warmstart_c12_9deg intel | ||
| cpld_restart_c12_9deg intel | ||
| control_CubedSphereGrid_parallel intel | ||
| control_latlon intel | ||
| control_wrtGauss_netcdf_parallel intel | ||
| control_c192 intel | ||
| control_c384 intel | ||
| control_c384gdas intel | ||
| control_stochy intel | ||
| control_stochy_restart intel | ||
| control_lndp intel | ||
| control_iovr4_gfdlmpv3 intel | ||
| control_p8 intel | ||
| control_p8.v2.sfc intel | ||
| control_p8_ugwpv1 intel | ||
| control_p8_ugwpv1_tempo intel | ||
| control_p8_ugwpv1_tempo_aerosol intel | ||
| control_p8_ugwpv1_tempo_aerosol_hail intel | ||
| control_restart_p8 intel | ||
| control_noqr_p8 intel | ||
| control_restart_noqr_p8 intel | ||
| control_decomp_p8 intel | ||
| control_2threads_p8 intel | ||
| control_p8_lndp intel | ||
| control_p8_rrtmgp intel | ||
| control_p8_mynn intel | ||
| merra2_thompson intel | ||
| merra2_hf_thompson intel | ||
| regional_control intel | ||
| regional_restart intel | ||
| regional_decomp intel | ||
| regional_2dwrtdecomp intel | ||
| regional_wofs intel | ||
| control_p8_rrtmgp_rad32 intel | ||
| rap_control intel | ||
| regional_spp_sppt_shum_skeb intel | ||
| rap_decomp intel | ||
| rap_2threads intel | ||
| rap_restart intel | ||
| rap_sfcdiff intel | ||
| rap_sfcdiff_decomp intel | ||
| rap_sfcdiff_restart intel | ||
| hrrr_control intel | ||
| hrrr_control_decomp intel | ||
| hrrr_control_2threads intel | ||
| hrrr_control_restart intel | ||
| rrfs_v1beta intel | ||
| rrfs_v1nssl intel | ||
| rrfs_v1nssl_nohailnoccn intel | ||
| control_csawmg intel | ||
| rap_unified_drag_suite_debug intel | ||
| regional_spp_sppt_shum_skeb_dyn32_phy32 intel | ||
| rap_control_dyn32_phy32 intel | ||
| hrrr_control_dyn32_phy32 intel | ||
| rap_2threads_dyn32_phy32 intel | ||
| hrrr_control_2threads_dyn32_phy32 intel | ||
| hrrr_control_decomp_dyn32_phy32 intel | ||
| rap_restart_dyn32_phy32 intel | ||
| hrrr_control_restart_dyn32_phy32 intel | ||
| rap_control_dyn64_phy32 intel | ||
| conus13km_debug_2threads intel | ||
| hafs_regional_atm intel | ||
| hafs_regional_atm_gfdlmpv3 intel | ||
| hafs_global_multiple_4nests_atm intel | ||
| hafs_regional_specified_moving_1nest_atm intel | ||
| hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 intel | ||
| hafs_regional_docn intel | ||
| hafs_regional_docn_oisst intel | ||
| datm_cdeps_control_cfsr intel | ||
| datm_cdeps_restart_cfsr intel | ||
| datm_cdeps_control_gefs intel | ||
| datm_cdeps_iau_gefs intel | ||
| datm_cdeps_stochy_gefs intel | ||
| datm_cdeps_ciceC_cfsr intel | ||
| datm_cdeps_mx025_cfsr intel | ||
| datm_cdeps_mx025_gefs intel | ||
| datm_cdeps_multiple_files_cfsr intel | ||
| datm_cdeps_3072x1536_cfsr intel | ||
| datm_cdeps_gfs intel | ||
| datm_cdeps_lnd_gswp3 intel | ||
| atm_ds2s_docn_pcice intel | ||
| atmaero_control_p8 intel | ||
| atmaero_control_p8_rad intel | ||
| atmaero_control_p8_rad_micro intel | ||
| cpld_regional_atm_fbh intel | ||
| control_c48 gnu | ||
| control_stochy gnu | ||
| control_p8 gnu | ||
| control_p8_ugwpv1 gnu | ||
| rap_control gnu | ||
| rap_decomp gnu | ||
| rap_2threads gnu | ||
| rap_restart gnu | ||
| rap_sfcdiff gnu | ||
| rap_sfcdiff_decomp gnu | ||
| rap_sfcdiff_restart gnu | ||
| hrrr_control gnu | ||
| hrrr_control_noqr gnu | ||
| hrrr_control_2threads gnu | ||
| hrrr_control_decomp gnu | ||
| hrrr_control_restart gnu | ||
| hrrr_control_restart_noqr gnu | ||
| rrfs_v1beta gnu | ||
| control_csawmg gnu | ||
| rap_noah_sfcdiff_cires_ugwp_debug gnu | ||
| rap_control_dyn32_phy32 gnu | ||
| hrrr_control_dyn32_phy32 gnu | ||
| rap_2threads_dyn32_phy32 gnu | ||
| hrrr_control_2threads_dyn32_phy32 gnu | ||
| hrrr_control_decomp_dyn32_phy32 gnu | ||
| rap_restart_dyn32_phy32 gnu | ||
| hrrr_control_restart_dyn32_phy32 gnu | ||
| conus13km_control gnu | ||
| conus13km_2threads gnu | ||
| conus13km_decomp gnu | ||
| conus13km_restart gnu | ||
| rap_control_dyn64_phy32 gnu | ||
| cpld_control_gfsv17 gnu | ||
| cpld_control_sfs gnu | ||
| datm_cdeps_control_cfsr gnu | ||
| control_gfs_mpas gnu | ||
| regional_wofs_tempo intel |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| ############################################################################### | ||
| # | ||
| # FV3_WoFS_v0 test atmosphere only with hi-res 3km | ||
| # | ||
| ############################################################################### | ||
|
|
||
| export TEST_DESCR="Compare FV3_WoFS_v0 hi-res 3km results with previous trunk version" | ||
|
|
||
| export CNTL_DIR=fv3_regional_wofs_tempo | ||
|
|
||
| export LIST_FILES="dynf000.nc \ | ||
| dynf006.nc \ | ||
| phyf000.nc \ | ||
| phyf006.nc \ | ||
| PRSLEV.GrbF00 \ | ||
| PRSLEV.GrbF06 \ | ||
| NATLEV.GrbF00 \ | ||
| NATLEV.GrbF06" | ||
|
|
||
| export_fv3_v16 | ||
|
|
||
| export FV3_RUN=regional_run.IN | ||
|
|
||
| export OZ_PHYS_OLD=.false. | ||
| export OZ_PHYS_NEW=.true. | ||
| export H2O_PHYS=.true. | ||
| export RESTART_INTERVAL="0" | ||
| export WRITE_DOPOST=.true. | ||
| export POSTAPP='lam' | ||
|
|
||
|
|
||
| export CCPP_SUITE=FV3_WoFS_v0_tempo | ||
| export FIELD_TABLE=field_table_thompson_aero_hail_tke | ||
| export IMP_PHYSICS=88 | ||
| export LTAEROSOL=.true. | ||
| export LTHAILAWARE=.true. | ||
| export DO_SAT_ADJ=.false. | ||
| export DIAG_TABLE=diag_table_aerosol.IN | ||
| export DIAG_TABLE_ADDITIONAL=diag_additional_hail | ||
|
|
||
| export INPUT_NML=regional_wofs.nml.IN | ||
| export MODEL_CONFIGURE=model_configure_regional.IN | ||
|
|
||
| export WLCLK=15 | ||
| export FHMAX=6 | ||
| export INPES=10 | ||
| export JNPES=11 | ||
| export OUTPUT_FH="3 -1" | ||
| WRITE_GROUP=1 | ||
| WRTTASK_PER_GROUP=10 | ||
| NTILES=1 | ||
|
|
||
| export IMP_PHYSICS=88 | ||
| export DNATS=0 | ||
| export DO_SAT_ADJ=.false. | ||
| export IALB=1 | ||
| export IEMS=1 | ||
| export HYBEDMF=.false. | ||
| export DO_MYNNEDMF=.true. | ||
| export DO_MYNNSFCLAY=.true. | ||
| export DO_DEEP=.false. | ||
| export SHAL_CNV=.false. | ||
| export IMFSHALCNV=-1 | ||
| export IMFDEEPCNV=-1 | ||
| export LSOIL_LSM=4 | ||
| export NSTF_NAME="2, 1, 0, 0, 0" | ||
|
|
||
| if [[ ${MACHINE_ID} = ursa ]]; then | ||
| export EXCLUSIVE_NODES=.true. | ||
| export MEM_PER_NODE="300G" | ||
| fi |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
where does this default MEM_PER_NODE
(used in Ursa #SBATCH --mem=@[MEM_PER_NODE])
come from?
Isn't this more than the 300G set in regional_wofs_tempo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default, Ursa gives 1.92G per core, and there are 192 cores/node. For a default TPN=192, the default memory should be 368640M. This memory amount is a system-level setting on Ursa. Note that Ursa has 2G per core available.
The reason why the tempo_p8 tests need 300G of memory is because they use 150 tasks, and by default, this does not provide enough memory for tempo.
see this comment: ufs-community/ccpp-physics#353 (comment)
If the regional_wofs_tempo test uses TPN=192, then it doesn't need the
export MEM_PER_NODE="300G"statement.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was kinda confused by the memory changes for Ursa too. With the addition of
export MEM_PER_NODE="368640M"in default_vars.sh, this is setting the memory for every Ursa test, right? Why is this needed if this is the same as the system-level setting on Ursa? Then, the individual TEMPO tests are asking forexport MEM_PER_NODE="300G", which is now less than the default. This is what is desired? Or do I have this wrong?I added the
export MEM_PER_NODE="300G"statement to regional_wofs_tempo. I thought it was necessary. Where are the number of tasks being set for the tempo_p8 tests that is different than the default?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the details, explanation, and sorting through the intermittent test failures
Does this fix for memory in one scheme practically change quite a few unrelated tests that also have TPN/=192?
If so, one option would be to comment this MEM_PER_NODE setting by default, similar to EXCLUSIVE_NODES_OPT or HIDE_UGWPV1 ideas
It would be more work to compute these based on system settings near compute_petbounds_and_tasks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes other tests with TPN/=192 would be changed. I don't know how to comment, or hide, a variable in the default settings though. But that would be the best solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One way is like AndersJensen-NOAA#3