Skip to content

Various fixes for CORDEX model data#3135

Merged
ghossh merged 36 commits into
mainfrom
more-cordex-fixes
Jul 14, 2026
Merged

Various fixes for CORDEX model data#3135
ghossh merged 36 commits into
mainfrom
more-cordex-fixes

Conversation

@bouweandela

@bouweandela bouweandela commented Jun 26, 2026

Copy link
Copy Markdown
Member

Description

Testing

These changes can be tested with recipe_perfmetrics_CORDEX-CMIP5.yml from ESMValGroup/ESMValTool#4199. It should also be possible to test the changes to the ensemble_statistics preprocessor function by using it in that recipe.


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.


To help with the number pull requests:

ghossh and others added 9 commits June 1, 2026 16:51
This update introduces a new section in the preprocessor documentation detailing how to regrid to standard CORDEX domains using the `target_grid` parameter. An example configuration for the `EUR-11` domain is provided, clarifying the use of domain names recognized by the `cordex` package.
@bouweandela bouweandela changed the base branch from main to target-grid-cordex-domain June 26, 2026 15:19
@bouweandela bouweandela added the fix for dataset Related to dataset-specific fix files label Jun 26, 2026
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.26%. Comparing base (46124b9) to head (f2b4988).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3135      +/-   ##
==========================================
+ Coverage   96.25%   96.26%   +0.01%     
==========================================
  Files         274      280       +6     
  Lines       16189    16258      +69     
==========================================
+ Hits        15582    15651      +69     
  Misses        607      607              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bouweandela bouweandela changed the title More cordex fixes Various fixes for CORDEX model data Jul 1, 2026
Base automatically changed from target-grid-cordex-domain to main July 2, 2026 19:58
@bouweandela bouweandela marked this pull request as ready for review July 2, 2026 20:35
@bouweandela bouweandela requested a review from ghossh July 2, 2026 20:38
@ghossh

ghossh commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

I got some of this issues from the recipe yesterday. I'm going to test this PR.

@bouweandela

Copy link
Copy Markdown
Member Author

Thanks! I'll stop working on this branch then and start a new branch for further fixes.

@ghossh

ghossh commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

You can keep making changes. I thought you finished when you put me in the reviewer. Just ping me again when you finish all changes.

@bouweandela

Copy link
Copy Markdown
Member Author

It's finished for the perfmetrics recipe, but then I tried adding another recipe plotting the models without first regridding and ran into #3145.

@ghossh

ghossh commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

I tried the recipe and it never finishes

26-07-12 10:03:50,282 UTC [506934] INFO    diagnostic/sst                [########################################] | 100% Completed | 222.17 s
2026-07-12 10:03:50,371 UTC [506934] INFO    Successfully completed task diagnostic/sst (priority 10)in 0:12:21.248472
2026-07-12 10:03:50,533 UTC [506696] INFO    Progress: 1 tasks running, 1 tasks waiting for ancestors, 9/11 done 

not progressing from there!! There is a problem with lst variable likely. Let me check further!.

@ghossh

ghossh commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

The recipe doesn't finish along with lst in 4hr. But it does work seperately but takes longer time.

@bouweandela

bouweandela commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

What settings are you using for Dask? The current recipe_perfmetrics_CORDEX-CMIP5.yml runs in about 25 minutes on Levante with these settings:

$ cat ~/.config/esmvaltool/dask.yml
max_parallel_tasks: 10
dask:
  use: threaded
  profiles:
    threaded:
      num_workers: 8

and produces this figure: ESMValGroup/ESMValTool#4199 (comment)
I did have issues with the distributed scheduler, saw a warning about a large task graph: I suspect that some of the preprocessor functions insert one or more large objects into the task graph.

@ghossh

ghossh commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

I was able to run the recipe successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix for dataset Related to dataset-specific fix files

Projects

Development

Successfully merging this pull request may close these issues.

2 participants