A description of validation processes and techniques for ocean forecasting

Sotillo, Marcos Garcia; Drevillon, Marie; Hernandez, Fabrice

doi:https://doi.org/10.5194/sp-5-opsr-16-2025

Articles | Volume 5-opsr

https://doi.org/10.5194/sp-5-opsr-16-2025

© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.

Report:

Ocean prediction: present status and state of the art (OPSR)

https://doi.org/10.5194/sp-5-opsr-16-2025

© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 5-opsr

02 Jun 2025

| OPSR | Chapter 6.2

| 02 Jun 2025 | OPSR | Chapter 6.2

A description of validation processes and techniques for ocean forecasting

Marcos Garcia Sotillo, Marie Drevillon, and Fabrice Hernandez

Download

Final revised paper (published on 02 Jun 2025)
Preprint (discussion started on 19 Sep 2024)

Interactive discussion

Status: closed

RC1:
'Comment on sp-2024-33', Anonymous Referee #1, 02 Oct 2024

The paper is a review of the different model validation carried out by different groups. Some model has more detailed information than others.

It is striking to note the amount of details given to the Canadian system. Similar details should be done on all the different systems listed in the present work as it will enrich the readers understanding of each system as well as provide useful information which will help other groups develop their system.
In term of usage of satellite data, (Line 173): are satellite data not used to validate model in the coastal region due to lack of remotely sensed data product or also due to higher uncertainty of remotely sensed coastal data?
Most important, I also suggest the authors include conclusion of their findings.
The following are minor correction:
Line 27: ETOOFS is not defined.
Line 60: Line EOV is not defined.
Line 104. I don't understand the sentence (may be missing something in the sentence)
Line 119: I suggest: "It aims to deliver a consistent, reliable, and state-of-the-art information derived from space..."
Line 132: PQ is not defined
Line 200: DA is not defined

Citation: https://doi.org/10.5194/sp-2024-33-RC1
- AC1:
  'Reply on RC1', Marcos Garcia Sotillo, 14 Feb 2025
  General Comments for Reviewers:
  The authors would like to express our sincere thanks for the insightful and constructive comments provided on the manuscript. All of the specific suggestions have been carefully considered and addressed in the revised version. Indeed, following their recommendations, the manuscript has been certainly improved.
  More specifically, the structure of the paper has been improved, enhancing its clarity, by means of including in the revised manuscript the following elements requested by Reviewers:
  A Paragraph on the goal of the paper, detailing its specific objectives.
  
  A Paragraph describing the Sections.
  
  A New Section on conclusions to give clearer take-home messages, and to ensure a summary of the discussion, reminding the reader of its relevance.
  
  As requested by both reviewers, the content of the validation of different regional systems has been homogenized, with additional details and references provided for some of them.
  Finally, the manuscript has been updated addressing all the specific comments and minor corrections proposed by both reviewers.
  We appreciate the time and effort the reviewers have put into evaluating our work. Their feedback has been invaluable in improving the manuscript.
  
  Specific comments for Reviewer 1.
  As requested, to reach similar level of detail than for the Canadian System, some extra details on model validations performed by other systems have been included.
  With respect to the question on the usage of satellite data (Line 173): “are satellite data not used to validate model in the coastal region due to lack of remotely sensed data product or also due to higher uncertainty of remotely sensed coastal data?”
  Both interpretations suggested by the reviewer are valid: There is a lack of remotely sensed data in the coastal zone (mainly due to sea-land boundary effects) and the available remote sense data in the coastal zone also exhibits a higher uncertainty there. The sentence has been revised in the manuscript for clarity.
  All minor corrections suggested by the Reviewer are included in the updated manuscript.
  
  Citation: https://doi.org/10.5194/sp-2024-33-AC1
RC2:
'Comment on sp-2024-33', Anonymous Referee #2, 08 Nov 2024

A description of Validation Processes and Techniques for Ocean Forecasting, Marcos G. Sotillo, Marie Drevillon, and Fabrice Hernandez
Summary
The manuscript reviews briefly the sources of observations used in operational ocean model systems before identifying some of the key variables used to perform different model validation. The last sections present the general evaluation framework in use for Copernicus Marine Service before presenting succinctly work done internationally outside of Copernicus.
General comments
The manuscript in its current form lacks clarity on multiple aspects.
Firstly, it is not obvious what are the specific objectives of the manuscript. Adding the typical paragraph(s) at the end of the introduction stating the main goals of the manuscript and the sections to achieve these objectives would significantly help orient the reader.
More importantly, the absence of a discussion / conclusions leaves the reader without a clear take-home message.
In its present status, the novelty and contribution of the manuscript is not clear to me. Review and strategic papers are generally useful to the community when they convey a clear message. The section on the Copernicus Marine Service, to my opinion, does not reflects the quality of the work done. CMEMS is often cited as the reference in terms of operational oceanography, and this is not clearly conveyed. A more detailed description of the procedures with examples and references would help illustrate this. The paper is too general and lacks references in its first sections while providing more references to previous work in the last section. I suggest adding to the Canadian experience a more comprehensive review of the validation work done in other groups internationally.
The syntax is overly complex and confusing with too many explanations between parentheses that disrupt the pace and obscure the message. An effort to simplify the writing would also help clarify the points brought up in the manuscript.
The use of specific jargon combined with missing definitions for multiple acronyms make the paper harder to read. I suggest reviewing the manuscript to define all acronyms at their first reference and propose the use of defined acronyms systematically.

Specific comments
L19. Define CMEMS as an acronym early on and use throughout the manuscript.
L27. ETOOFS not defined
L31. Mention clearly the overall objectives and short descriptions of the manuscripts sections and how it helps achieve it.
L37. What is mostly leaving out? Please clarify
L38. Unclear please rephrase the beginning of the sentence.
L47. Define MFC NRT
L52. I would make the sentence between (…) a separate sentence.
L60. EOV not defined
L63. PQ not defined
L63. strategic plan points to what? Defining what the following points are would clarify, limitations, gaps, etc.
L67. Separate sentences at ;
L71. Explicitly mention the limitation for clarity
L82. This section should present more clearly temperature in general before addressing the SST and deeper temperature evaluation. Currently the section is presented as SST, making the Argo discussion look out of place.
L94. Sea level should be a separate point
L97-101. Make these two sentences for clarity.
L109-115. Revise and clarify. Avoid using too many parentheses and verify all opened parentheses are closed.
L110-111 avoid unnecessary repetitions, in this case three OOFS in two sentences. Applies elsewhere in the manuscript.
L117. See previous comment on systematically using CMEMS.
L118. Rephrase complex system of systems framework
L137. Provide examples and/or references here
L144. consistency across what? the different OOFS? Please clarify.
L154. OOFS repetition
L176. Broken link
L188. No opening parentheses
L190. Should be … in house…
L195-200 Any references for this part of the discussion, a project or website documenting those short-lived models? This application is similar to some discussed relocatable coastal ocean system. More details on the methodology would also be interesting as such applications would increase in the coming years.
L210. Reference to Paquin et al. should be corrected in the text & the references. Please use :
Paquin, JP., Roy, F., Smith, G.C., MacDermid, S., Lei, J., Dupont, F., Lu, Y., Taylor, S., St-Onge-Drouin, S., Blanken, H., Dunphy, M., Soontiens, N. A new high-resolution Coastal Ice-Ocean Prediction System for the East Coast of Canada. Ocean Dynamics 74, 799–826 (2024). https://doi.org/10.1007/s10236-024-01634-7
L210. Paquin et al. 2019 (see reference below) presented the prototype of the mentioned port models. Nudds et al. (2020) presents the initial intercomparison projects that took place to compare the NEMO model implementation described in Paquin et al. (2019) with an unstructured model implementation using FVCOM. Please consider adding.
Paquin J-P, Lu Y, Taylor S, Blanken H, Marcotte G, Hu X, Zhai L, Higginson S, Nudds S, Chanut J, Smith GC, Bernier N, Dupont F (2019) High-resolution modelling of a coastal harbour in the presence of strong tides and significant river runoff. Ocean Dyn 70:365–385. https://doi.org/10.1007/s10236-019-01334-7
Nudds S, Lu Y, Higginson S, Haigh SP, Paquin J-P, O’Flaherty-Sproul M, Taylor S, Blanken H, Marcotte G, Smith GC, Bernier NB, MacAulay P, Wu Y, Zhai L, Hu X, Chanut J, Dunphy M, Dupont F, Greenberg D, Davidson FJM, Page F (2020) Evaluation of structured and unstructured models for application in operational ocean forecasting in Nearshore Waters. J Mar Sci Eng 8(7):484. https://doi.org/10.3390/jmse8070484
L217. please replace “…” by “etc.”
L224. Here again references or website would increase impact of argument. Otherwise the discussion remains to vague to really be helpful.
L233. The manuscript needs a conclusion to summarize the discussion and remind the reader of the relevance of the discussion.

References - Review carefully references and standardize format
L284. Inconsistency Mourre et al. 2021 in text, 2018 in reference
L291. Correct Paquin et al. to provided reference
L318. Inconsistency Sotillo 2019 in reference, 2020 in text.

Citation: https://doi.org/10.5194/sp-2024-33-RC2
- AC2:
  'Reply on RC2', Marcos Garcia Sotillo, 14 Feb 2025
  General Comments for Reviewers:
  The authors would like to express our sincere thanks for the insightful and constructive comments provided on the manuscript. All of the specific suggestions have been carefully considered and addressed in the revised version. Indeed, following their recommendations, the manuscript has been certainly improved.
  More specifically, the structure of the paper has been improved, enhancing its clarity, by means of including in the revised manuscript the following elements requested by Reviewers:
  A Paragraph on the goal of the paper, detailing its specific objectives.
  
  A Paragraph describing the Sections.
  
  A New Section on conclusions to give clearer take-home messages, and to ensure a summary of the discussion, reminding the reader of its relevance.
  
  As requested by both reviewers, the content of the validation of different regional systems has been homogenized, with additional details and references provided for some of them.
  Finally, the manuscript has been updated addressing all the specific comments and minor corrections proposed by both reviewers.
  We appreciate the time and effort the reviewers have put into evaluating our work. Their feedback has been invaluable in improving the manuscript.
  
  Specific comments for Reviewer 2.
  The paper has been extensively revised in accordance with the reviewer’s suggestions. The updated manuscript includes an enhanced discussion and a clearer explanation of the relevance of the proposed review.
  
  An effort to improve the writing has been made to clarify the point brought up, simplifying syntaxis and reducing the use of brackets in the revised manuscript, as suggested by the Reviewer.
  
  The use of specific model validation jargon, and acronyms, has been minimized to ease the paper readability. As suggested, all acronyms have been systematically defined at the first reference.
  
  “The section on the Copernicus Marine Service, to my opinion, does not reflects the quality of the work done. CMEMS is often cited as the reference in terms of operational oceanography, and this is not clearly conveyed. A more detailed description of the procedures with examples and references would help illustrate this.”
  
  As requested, the section on Copernicus Marine has been updated to better reflect the service’s work in model validation. As stated in the updated manuscript (Section 3), Copernicus Marine is a benchmark in operational oceanography, offering a comprehensive multi-product service portfolio. Its catalog includes over 150 operational global/regional products and covers more than 60 essential ocean variables across the blue, green, and white ocean.
  As discussed in the paper and highlighted in the new conclusion section, the service operates through a distributed production network that connects several regional production centers. These centers run models for ocean physics, including sea ice and wave modeling systems, as well as biogeochemistry. The service ensures the delivery of homogenized product quality information across all variables and products offered. To achieve this, the Copernicus Marine Service organizes product quality information from producers, providing dedicated scientific documentation that is then communicated to end-users. Every product in the Copernicus Marine portfolio is accompanied by the relevant product quality documents, and updated estimated accuracy values for every product are made available through the Copernicus Product Quality Dashboard.
  The manuscript also refers to the service's extensive operational experience and to the analysis of strengths and weaknesses of the ongoing process to establish a common, homogeneous Product Quality framework across Copernicus Marine products and EOVs.
  About the Reviewer 2 Specific comments:
  
  All the specific comments made by the Reviewer have been addressed by the authors in the updated manuscript.
  The authors state that the only exception where the authors did not follow the Reviewer’s recommendation concerns the point: “L19. Define CMEMS as an acronym early on and use throughout the manuscript.”
  With this respect, the authors note that the Copernicus Marine Service recently distributed among their producers a guideline recommending the avoidance of acronyms (e.g., CMEMS, CMS) to refer to the service. As a result, the CMEMS acronym is no longer used throughout the revised manuscript, except when referencing papers or reports from the time when the service was referred to by that acronym.
  
  Citation: https://doi.org/10.5194/sp-2024-33-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Publish subject to minor revisions (review by editor) (17 Feb 2025) by Enrique Álvarez Fanjul

AR by Marcos Garcia Sotillo on behalf of the Authors (21 Feb 2025) Author's response Author's tracked changes Manuscript

ED: Publish as is (25 Feb 2025) by Enrique Álvarez Fanjul

AR by Marcos Garcia Sotillo on behalf of the Authors (04 Mar 2025) Manuscript

Short summary

Operational forecasting systems require best practices for assessing the quality of ocean products. The authors discuss the role of the observing network in performing validation of ocean models, identifying current gaps but also emphasizing the need of new metrics. An analysis on the level of maturity of validation processes from global to regional systems is provided. A rich variety of approaches exists. An example is provided of how the Copernicus Marine Service organizes product quality information.