the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 07 Oct 2024
A description of Model Intercomparison Processes and Techniques for Ocean Forecasting
Abstract. The availability of numerical simulations for ocean past estimates or future forecast worldwide at multiple scales is opening new challenges in assessing their realism and predictive capacity through an intercomparison exercise. This requires a huge effort in designing and implementing a proper assessment of models’ performances, as already demonstrated by the atmospheric community that was pioneering in that sense. Historically, the ocean community launched only in the recent period dedicated actions aimed at identifying robust patterns in eddy-permitting simulations: it required definition of modelling configurations, execution of dedicated experiments that deal also with the storing of the outputs and the implementation of evaluation frameworks. Starting from this baseline, numerous initiatives like CLIVAR for climate research and GODAE for operational systems have raised and are actively promoting best practices through specific intercomparison tasks, aimed at demonstrating the efficient use of the Global Ocean Observing System and the operational capabilities, sharing expertise and increase the scientific quality of the numerical systems. Examples, like the ORA-IP, or the Class 4 near real time GODAE intercomparison are introduced and commented, discussing also on the ways forward on making this kind of analysis more systematic for addressing monitoring of ocean state in operations.
- Preprint
(2249 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on sp-2024-39', Anonymous Referee #1, 10 Nov 2024
This paper I believe is intended as a brief review of past efforts and methods to compare ocean state and ocean forecast products that have been developed within the international community. While this is a useful objective I find the current version of the paper has many problems with it especially if read on its own. I understand that it would form 1 chapter of a larger report but I do suggest that someone should be reviewing the report as a whole
The paper does reference other “Chapters” and sometimes non-existent “sections” Lines29,61,66,84,100. These need to be properly checked
The text also has lots of acronyms and other notation that will mean nothing to a wider audience eg ET/OOFS, Class 4, L4 products, “go-no-go”?. I appreciate these may appear in other chapters but they should at least be defined here or cross referenced when first mentioned.
The AMIP concept is rightly introduced and a valuable concept, but seems odd then not to mention OMIP? And then finally leading to CMIP. The evolution of objectives to define the actual ocean states should then refer to the success of ERA and other atmospheric reanalyses. Emphasising the different objectives of GODAE and CLIVAR in comparing reanalyses for the ocean could then be explained and would then follow naturally. State estimation and forecasting as different applications.
Section 3.1 should start by properly justifying the value of observation space verification. The issue of independent v non-independent data comparisons should be discussed. Different kinds of metrics and their usefulness would be useful to summarise more clearly here.
Section 3.2 good topic to discuss the value of ensemble comparisons but it does not really discuss the value of ensemble product. Are biases in individual products reduced in this way? Uotila et al 2019 polar comparison is an example of this. L154 top right panel seems wrong? Include reference for Fig 1 legend.
Sections 3.3, It would be nice if this issue of regional studies was brought more up to date? The references are all rather old? These and the following 2 sections are very brief
3.4 This section is so brief and in no way lives up to the promise of the long heading. What are the key points?
3.5 Again very brief and the title seems to promise a future look but this is entirely absent.
I would say this paper needs more attention before it should, be published. It needs more careful reading through and some thought given to making it more accessible for the wider audience. For State of the Planet is “expert-based assessments of academic findings curated for a wider audience to support decision making”
Citation: https://doi.org/10.5194/sp-2024-39-RC1 -
AC1: 'Reply on RC1', Fabrice Hernandez, 10 Jan 2025
This paper I believe is intended as a brief review of past efforts and methods to compare ocean state and ocean forecast products that have been developed within the international community. While this is a useful objective I find the current version of the paper has many problems with it especially if read on its own. I understand that it would form 1 chapter of a larger report but I do suggest that someone should be reviewing the report as a whole
The paper does reference other “Chapters” and sometimes non-existent “sections” Lines29,61,66,84,100. These need to be properly checked
Initially this article is a « chapter » or « section » as part of the report published in State of the Planet: « Ocean prediction: present status and state of the art ». Inside this report, this article is strongly associated with the one entitled: «A description of Validation Processes and Techniques for Ocean Forecasting » . The present article is going to be modified in order to change « chapter » with the exact citation. Like:
Sotillo, M. G., Drevillon, M., and Hernandez, F.: A description of Validation Processes and Techniques for Ocean Forecasting, State Planet Discuss. [preprint], https://doi.org/10.5194/sp-2024-33, in review, 2024.
—> changed at line 24 with: Garcia-Sotillo et al. (2024, this report).
—> changed at line 61 « (mentioned in Section 4.2.3) » by « (mentioned in Ardhuin et al., 2024, this report) «
—> changed at line 66 « also referenced in the chapter 2.15 above« by « referenced in this report (Garcia-Sotillo et al., 2024)«
—> changed at line 84 «(see Section 4.3.2) above « by « see Garcia-Sotillo et al. (2024) in this report for «
—> changed at line 100 «as reminded in Section 4.3.2« by « as reminded 100 in Section 4.3.2 «
The text also has lots of acronyms and other notation that will mean nothing to a wider audience eg ET/OOFS, Class 4, L4 products, “go-no-go”?. I appreciate these may appear in other chapters but they should at least be defined here or cross referenced when first mentioned.
Agreed, even if defined in previous « chapters » of the report, we propose to explicitly define acronyms or « jargon » :
—> changed at line 61: « WMO » by « World Meteorological Organization (WMO) «
—> changed at line 63: « GODAE » by « Global Ocean Data Assimilation Experiment (GODAE) «
—> changed at line 68: « OOFS » by « operational ocean forecasting system (OOFS)«
—> changed at line 74: « ocean predict » by « Ocean Predict (https://oceanpredict.org/) )«
—> changed at line 75: « ETOOFS » by « the Expert Team on Operational Ocean Forecasting Systems (ETOOFS )«
—> changed at line 105: « CMEMS » by «Coperrnicus Marine Environment Monitoring Service (CMEMS) «
—> changed at line 139: « when referring to L4 observation products » by «when referring to re-processed/re-gridded observation products (also called Level 4 or L4 type of observed products). «
For « class1 » etc …. the definition is given in chapter Garcia-Sotillo et al., 2024, and it is not reproduced here. We propose the following modification at line 65:
« The preliminary task was to define the validation concepts and methodologies (Hernandez et al., 2015a), with the so called “ Class 1 to 4 metrics” described in this report (Garcia-Sotillo et al., 2024), and that directly inherited from the weather forecast verification methods (Murphy, 1993). «
The expression « go/no-go » is self explicit and used by operational teams to decide or not to carry on some action, in this case, put in operation the new system. We propose to keep it at it is.
« NetCDF » is also commonly used in oceanography. We propose to modify line 88 to : « NetCDF file format «
The AMIP concept is rightly introduced and a valuable concept, but seems odd then not to mention OMIP? And then finally leading to CMIP. The evolution of objectives to define the actual ocean states should then refer to the success of ERA and other atmospheric reanalyses. Emphasising the different objectives of GODAE and CLIVAR in comparing reanalyses for the ocean could then be explained and would then follow naturally. State estimation and forecasting as different applications.
We would like to thank the reviewer for offering this insight, which we had not initially considered, in our desire to deal directly and uniquely with intercomparisons for operational oceanography. From line 34 of our first section a full development is proposed to consider CMIP and OMIP efforts, mentioning the CORE reference framework and the recent CMIP6 experiments.
Section 3.1 should start by properly justifying the value of observation space verification. The issue of independent v non-independent data comparisons should be discussed. Different kinds of metrics and their usefulness would be useful to summarise more clearly here.
Although Class4 metrics concept is reminded in Garcia-Sotillo et al. (2024) with many past references, we propose to add at the beginning of section 3.1 an introduction of the Class4 metrics, and also introduce the « independent/non-independent » observation impact on the Class4 evaluation.
Section 3.2 good topic to discuss the value of ensemble comparisons but it does not really discuss the value of ensemble product. Are biases in individual products reduced in this way? Uotila et al 2019 polar comparison is an example of this. L154 top right panel seems wrong? Include reference for Fig 1 legend.
We agree that ensemble forecast needed to be better introduced. We propose in section 3.2 to describe first ensemble forecast initiatives then going to multi-model inter comparison in terms of ensembles through the dedicated exemple of Figure 1.
There is no reference for Fig 1 because we have produced this original figure from products extracted at the Copernicus Marine and Climate Service DataStore. We propose to upgrade the figure (now from 1980 to 2024) with a more explicit representation of the ENSEMBLE average from all reanalyses, then discuss the merit of this ensemble in section 3.2. The figure caption of Figure 1 is changed with more explicit description, because we propose to change the upper/middle left panels of Figure 1: we introduce on the top panel the differences of each product against the ENSEMBLE mean, and we move the former top panel (Box-averaged SST anomalies relative to a common climatology, called also the « SST index ») at the middle. Instead of using the OSTIA SST as a reference in the statistics of the Taylor Diagram, we use ARMOR3D, that offers longer time consistency (product till November 2024). With upgraded Figure 1, we propose to fully change the text of this section 3.2, with a reference to Uotila et al (2019) with the assessment of the ENSEMBLE estimate.
Sections 3.3, It would be nice if this issue of regional studies was brought more up to date? The references are all rather old? These and the following 2 sections are very brief
We agree with the review and propose for this section an extended discussion with recent references to inter comparison at regional scales, considering global versus regional system intercomparison. We also remind that inter comparison at regional scales depends on the reduced number of regional systems that overlap into a given area, and we discuss the importance of assessing the error propagation from « parents » model that feed boundary conditions for « child » regional models.
3.4 This section is so brief and in no way lives up to the promise of the long heading. What are the key points?
This section aims at describing the past, present and future effort on intercomparing ocean reanalyses produced by operational centres. Then suggest areas of improvements for the intercomparison activities. We propose to highlight more specifically Key points and to provide hints for future inter-comparison efforts.
3.5 Again very brief and the title seems to promise a future look but this is entirely absent.
We propose to change the title of the section by: « A perspective of ocean reanalyses intercomparison: the ocean state monitoring «
Then, keep the description and reference of Ocean State Monitoring. This is short, but we do not see much more to say on it.
I would say this paper needs more attention before it should, be published. It needs more careful reading through and some thought given to making it more accessible for the wider audience. For State of the Planet is “expert-based assessments of academic findings curated for a wider audience to support decision making”
We fully agree with the idea to provide an article for a large audience. Our propositions above are made in this sense: less acronyms, more explicit text, and dedicated references for more in-depth interested readers.
Citation: https://doi.org/10.5194/sp-2024-39-AC1
-
AC1: 'Reply on RC1', Fabrice Hernandez, 10 Jan 2025
-
RC2: 'Comment on sp-2024-39', Anonymous Referee #2, 15 Nov 2024
This article presents an historical and exhaustive review of the different intercomparison exercises done in the Ocean Forecasting community, including ocean reanalysis intercomparison exercises. Methods and outcomes are discussed.
The paper is well-written, but I would suggest few improvements to make it easier to understand to readers outside the operational forecasting community. There are many acronyms that should be defined the first time they appear in the text (GODAE, CLIVAR, OOFS, CMEMS, ORA-IP, ETOOFS) and some “internal vocabulary” that can be explained in common words as class1, class2 and class3.
The addition of a concluding section highlighting the challenges and opportunities that are coming with, for example, the ensemble approaches for analysis and forecasts, the higher resolution system with an increased data volume to handle and intercomparison methods based on machine learning would make the paper more impactful even if mentioned previously in the different sections.
The problem of the double penalty when comparing products at different resolution is not mentioned. This can be done in section 2.2, when discussing the representativity or 3.1 with the class4. It is also related to the regridding approach used in some intercomparison projects.
I found the section 3.2 confusing. First ensemble approach is related to forecast ensemble, from the same system, but then ensemble is related to an ensemble of forecasts coming from different OOFs. Intercomparison exercises will offer more opportunities if involving ensemble forecasts, with possible comparison of the different spread characteristics.
In section 3 describing intercomparison exercises in different context, I would suggest adding the UN Decade SynObs project, intercomparing different Observing Systems Experiments (OSE) to assess the impact of diverse ocean observing systems on different OOFs.
Line by line comments
l.62: I would suggest adding “ocean operational forecasting system” to differentiate from other ocean operational products based only on observations and from line 49 dealing with the 1st intercomparison also but for ocean reanalysis.
l.80: the spread of the ensemble is also used as an uncertainty estimation. In the atmospheric community it may be more often seen as a reference (verification against analysis) which is less common in the oceanographic community.
l.92: OOFS or OOFSs?
l.106 to 109: You may refine the potential use of those emerging methods. This could also be addressed in a conclusion section.
l.114: to compare discrete observations?
l.131: scales and processes even for observations, especially the remote ones that are the result of complex treatments.
l.159: the definition/examples of class1 metric should be given, for example: daily 2D and 3D model fields on a common grid.
Figure 1: legends are very small.
l.176: can you describe in few words the tools developed?
Citation: https://doi.org/10.5194/sp-2024-39-RC2 -
AC2: 'Reply on RC2', Fabrice Hernandez, 10 Jan 2025
This article presents an historical and exhaustive review of the different intercomparison exercises done in the Ocean Forecasting community, including ocean reanalysis intercomparison exercises. Methods and outcomes are discussed.
The paper is well-written, but I would suggest few improvements to make it easier to understand to readers outside the operational forecasting community. There are many acronyms that should be defined the first time they appear in the text (GODAE, CLIVAR, OOFS, CMEMS, ORA-IP, ETOOFS) and some “internal vocabulary” that can be explained in common words as class1, class2 and class3.
We have try to be more explicit in the text, and define all acronyms, and give more explanations considering the technical vocabulary like « Class 4 »
The addition of a concluding section highlighting the challenges and opportunities that are coming with, for example, the ensemble approaches for analysis and forecasts, the higher resolution system with an increased data volume to handle and intercomparison methods based on machine learning would make the paper more impactful even if mentioned previously in the different sections.
We propose to up-grade the end of section 3.4 with specific considerations on intercomparison framework and ensemble forecast. And we transform section 3.5 in perspective of intercomparison for Ocean Climate/State Monitoring
The problem of the double penalty when comparing products at different resolution is not mentioned. This can be done in section 2.2, when discussing the representativity or 3.1 with the class4. It is also related to the regridding approach used in some intercomparison projects.
We thank the reviewer to raise the double penalty effect on verification procedures. We propose to mention this aspect in both sections 2.2 and 3.1 when discussing representativity issues.
I found the section 3.2 confusing. First ensemble approach is related to forecast ensemble, from the same system, but then ensemble is related to an ensemble of forecasts coming from different OOFs. Intercomparison exercises will offer more opportunities if involving ensemble forecasts, with possible comparison of the different spread characteristics.
We propose to correct this section: with at the beginning an introduction to ensemble forecast from individual operational systems, then moving to multi-system inter comparisons through the specific example proposed in Figure 1.
In section 3 describing intercomparison exercises in different context, I would suggest adding the UN Decade SynObs project, intercomparing different Observing Systems Experiments (OSE) to assess the impact of diverse ocean observing systems on different OOFs.
We thank the reviewer for this valuable comment. We propose to introduce the SYNOBS project in section 3.1, with recent references
Line by line comments
l.62: I would suggest adding “ocean operational forecasting system” to differentiate from other ocean operational products based only on observations and from line 49 dealing with the 1st intercomparison also but for ocean reanalysis.
We propose to re-write partly this section
l.80: the spread of the ensemble is also used as an uncertainty estimation. In the atmospheric community it may be more often seen as a reference (verification against analysis) which is less common in the oceanographic community.
We propose to re-write partly this section
l.106 to 109: You may refine the potential use of those emerging methods. This could also be addressed in a conclusion section.
l.114: to compare discrete observations?
We propose now to emphasize IA derived techniques more carefully with ad-hoc references in our chapter 3
l.131: scales and processes even for observations, especially the remote ones that are the result of complex treatments.
We propose to address more in depth issues on scales and representativity of observed products, when used as reference for inter comparison .
l.159: the definition/examples of class1 metric should be given, for example: daily 2D and 3D model fields on a common grid.
We propose to exemplify Class1 metrics
Figure 1: legends are very small.
We propose to reprocess and upgrade Figure 1 . Doing so, we gathered data in the Copernicus Marine and Climate Service DataStore until november 2024
l.176: can you describe in few words the tools developed?
We propose to complement the text using Lorente et al., (2019) reference.
Citation: https://doi.org/10.5194/sp-2024-39-AC2
-
AC2: 'Reply on RC2', Fabrice Hernandez, 10 Jan 2025
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 90 | 32 | 107 | 229 | 6 | 4 |
- HTML: 90
- PDF: 32
- XML: 107
- Total: 229
- BibTeX: 6
- EndNote: 4
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1