A description of model intercomparison processes and techniques for ocean forecasting

The availability of numerical simulations for ocean dynamics past estimates or future forecast worldwide at multiple scales is opening new challenges in assessing their realism and predictive capacity through an intercomparison exercise. This requires a huge effort in designing and implementing a proper assessment of models' performances, as already demonstrated by the atmospheric community that was pioneering in that sense. Historically, the ocean community has only recently launched dedicated actions aimed at identifying robust patterns in eddy-permitting simulations: it required definition of modelling configurations, execution of dedicated experiments that also deal with the storing of the outputs and the implementation of evaluation frameworks. Starting from this baseline, numerous initiatives like WCRP/Climate Variability and Predictability (CLIVAR) for climate research and the Global Ocean Data Assimilation Experiment (GODAE) for operational systems have arisen and are actively promoting best practice through specific intercomparison tasks, aimed at demonstrating the efficient use of the Global Ocean Observing System and its operational capabilities, sharing expertise, and increasing the scientific quality of the numerical systems. Examples, like the Ocean Reanalysis Intercomparison Project (ORA-IP) or the Class 4 near-real-time GODAE intercomparison, are introduced and commented on, also discussing ways forward for making this kind of analysis more systematic using artificial intelligence approaches for addressing monitoring of ocean state in operations or facilitating in-house routine verification in ocean forecasting centres.

1 Historical development of model intercomparisons

Historically, in oceanography, model comparisons began with evaluations of “free” and “forced” numerical simulations of ocean circulation over the same space and time frames, assessing their differences within comparable situations. The international Atmospheric Model Intercomparison Project (AMIP), under the World Climate Research Programme (WCRP), played a pioneering role in guiding the oceanic modelling community (Gates, 1992). AMIP's primary objective was to comprehensively evaluate each model's performance and document systematic errors. From an academic standpoint, this intercomparison aimed to identify avenues for enhancing future atmospheric models and driving further developments. Consequently, this approach aligns clearly with the validation framework outlined in Sotillo et al. (2025, in this report). To provide an objective assessment of each “competing” model's performance, a common “reference truth” was selected, such as climatology or atmospheric reanalysis (deemed more realistic than the AMIP simulations). This process involved analysing a series of targeted key variables extracted from the model state to provide an overview of the model's skill in representing various atmospheric aspects.

In 1996, the same atmospheric community, involved in climate studies, settled the basis of the Coupled Model Intercomparison Project (CMIP) under the auspice of the WCRP/Climate Variability and Predictability (CLIVAR) panel to document systematic errors of global couple climate simulations in support of the Intergovernmental Panel on Climate Change (IPCC) framework (Meehl et al., 1997). Over the six phases of the CMIP, intercomparisons have refined the assessments, increasingly including the physical, biochemical and ecosystem components of the Earth system, by testing various climate scenarios of past, present and future CO₂ emissions. In the current phase, the CMIP6, the variety of models, simulations and their objectives have led the community to redefine the federated structure through a common specific framework, the Diagnostic, Evaluation and Characterization of Klima (DECK) experiments, which set out the simulations and scientific questions to be addressed. The DECK is the new acceptance criterion for a climate intercomparison project in the CMIP (Eyring et al., 2016). The evolution of the CMIP has been accompanied by the gradual adoption by the climate community of common standards, coordination, infrastructure and documentation, accessible to all. This persistent framework aims to ensure continuity in climate model performance assessment of future CMIP phases in which re-processed historical simulations defined in the AMIP would allow changes and benefits of more elaborated components of the Earth system models (ESMs).

The ocean modelling research community adopted a similar approach to the AMIP when the first global- or basin-scale eddy-permitting ocean simulations were achieved in the 1990s. The US–German Community Modelling Effort (CME), in support of the World Ocean Circulation Experiment (WOCE), started to infer model parametrization and sensitivity studies in modelling the North Atlantic basin (Böning and Bryan, 1996). Sources of errors like ocean boundaries or vertical mixing parametrization were identified. The DYNAMO project, dedicated to offering intercomparison among three classes of ocean models of the North Atlantic Ocean in a similar numerical experiment framework (Meincke et al., 2001), allowed patterns of the North Atlantic Ocean circulation to be identified that were robust and other patterns that were sensitive to model parametrization. In this case, the intercomparison approach brought another benefit than just identifying performances among the simulations: the common and matching patterns represented by the simulations were considered updated knowledge of the North Atlantic circulation. In other terms, the “ensemble pattern” from the simulations is identified as a robust representation of the “ocean truth” at the scales simulated by these models.

This first initiative led to the development of a common ocean modelling framework from the ocean community also involved in the CMIP projects, the Coordinated Ocean-ice Reference Experiments (COREs), aiming to provide common references for consistent assessment from a multi-model perspective (Griffies et al., 2009). CORE-I intends to evaluate model mean biases under a normal year forcing, using a prescribed series of metrics (e.g. Danabasoglu et al., 2014). The CORE-II framework extends the ocean model evaluation under the common interannual forcing – starting in 1948 – proposed initially by Large and Yeager (2009). It offers more direct comparison to ocean observations and to the effective ocean interannual variability. An intercomparison of 18 time-dependent ocean numerical simulations have been performed so far, with useful outcomes for global ocean model improvements. The CORE-II approach is the foundation of the Ocean Model Intercomparison Projects (OMIPs) carried out in support of the successive CMIPs, with a coordinated evaluation of the ocean, sea ice, tracer and biogeochemistry simulations forced by common atmospheric datasets (Eyring et al., 2016). The OMIP version 1 contribution to CMIP6, with ocean simulations' intercomparisons over the 1948–2009 period, is described by Griffies et al. (2016) and contains a comprehensive list of metrics and guidance to evaluate ocean–sea ice model skills as part of ESMs. A companion article by Orr et al. (2017) proposes the evaluation framework for the biogeochemical coupled model simulations in CMIP6. Under the CLIVAR Ocean Model Development Panel (OMDP) coordination, OMIP version 2 is ongoing using the more recent JRA-55 reanalysis forcings (Kobayashi et al., 2015). Metrics of the ocean (equivalent here to diagnostics) endorsed by the OMIP are those recommended for the assessment of ocean climate behaviour, impacts and scenarios in the CMIP DECK.

These first ocean intercomparison projects witness the community effort, trying to commonly define modelling strategies; conduct the simulations individually; and then intercompare the simulations in order to evaluate the model's performance with regard to observed realistic references. The projects bring better characterization of model errors and weaknesses considering specific ocean processes, from physical to biogeochemical aspects, over decadal, interannual and seasonal timescales. Implicitly, these efforts have involved strategies for distributing, storing and sharing simulations and metrics, under constraints of computer server limitations in capacity and communication bandwidth. In practice, this added to the common technical definition of standards shared by all participants and a fitness-for-purpose evaluation framework to be applied in similar ways for every simulation. And finally, a common synthesis effort is carried out in order to provide valuable conclusions.

The first intercomparison project that involved the operational oceanography has been carried out in the frame of the CLIVAR Global Synthesis and Observation Panel (GSOP). In practice, this involved intercomparing different ocean reanalyses computed over several decades and providing “ocean synthesis” on ocean state estimation through a chosen series of essential ocean variables (EOVs) considered in climate research (Stammer et al., 2009). A step was taken since it was no longer comparison of model outputs but of products issued from the more complex system producing each reanalysis (observation + model + assimilation), increasing the factors of discrepancies among them. The idea is that multi-system ensemble approaches should be useful to obtain better estimates of the ocean evolution. The GSOP objectives were (1) to assess the consistency of the synthesis through intercomparison; (2) to evaluate the accuracy of the products, possibly by comparison to observations; (3) to estimate uncertainties; (4) to identify areas where improvements were needed; (5) to evaluate the lack of assimilated observations that directly impacted the synthesis and propose future observational requirements; and (6) to work on new approaches, like coupled data assimilation. One of the outcomes was to highlight common behaviour among some products, that is, evidence “clusters” and correlated patterns that sometimes had just inappropriate biases.

In the atmospheric and weather-forecast side, usually responsible for marine meteorology predictions, routine intercomparison for wave forecast has been settled for many years under the World Meteorological Organization (WMO) framework. The European Centre for Medium-Range Weather Forecasts (ECMWF) hosts the ongoing WMO Lead Centre for Wave Forecast Verification where 18 regional and global wave forecast systems are compared (https://confluence.ecmwf.int/display/WLW, last access: 29 January 2025). Beyond wave forecasts' verification and quality monitoring, the ECMWF commits to maintaining an archive of the verification statistics to allow the generation and display of trends in performance over time.

A first dedicated intercomparison of ocean operational systems, operated on routine, was achieved by the Global Ocean Data Assimilation Experiment (GODAE) community (Bell et al., 2009), through an intercomparison of hindcasts over 2008. The main objectives were to (a) demonstrate GODAE operational systems in operations, (b) share expertise and design validation tools and metrics endorsed by all GODAE operational centres, and (c) evaluate the overall scientific quality of the different GODAE operational systems. The preliminary task was to define the validation concepts and methodologies (Hernandez et al., 2015a), with the so-called “Class 1 to 4 metrics” described in this report (Sotillo et al., 2025), and those directly inherited from the weather forecast verification methods (Murphy, 1993). A demanding task was to provide similar “Class 1”, “Class 2” and “Class 3” files from each Operational Ocean Forecasting System (OOFS) and then to carry out the evaluation through intercomparison and validation against “truth references” (Hernandez, 2011).

2 Key findings for state-of-the-art model intercomparison

2.1 From academia to operation: adoption of best practice

The legacy of the first 10 years of GODAE was the implementation of an expert community for OOFS intercomparison: the Intercomparison and Validation Task Team (IVTT). This group was created during GODAE, continuing its activity in GODAE OceanView and, up to present day, in Ocean Predict (https://oceanpredict.org/, last access: 29 January 2025). A second benefit was the development of an ad hoc validation and intercomparison methodology, improved and tested regularly since, until it was adopted as best practice and recommended by the Expert Team on Operational Ocean Forecasting Systems (ETOOFS; Alvarez-Fanjul et al., 2022).

As a result of these activities, it was found that performing intercomparison of OOFSs and models brought the following aspects to address:

• Characterize the performance of individual OOFSs of the same kind relatively to a given “truth”, identify outliers and give clues for further OOFS improvements.

• Allow “ensemble estimation” that provides qualitatively more robust and reliable estimates, i.e. the “ensemble mean” approach. In some cases, if the “ocean truth” is missing, the ensemble mean can be considered a reference and be used to validate individually the systems.

• Provide an ad hoc methodology for operational qualification; see Sotillo et al. (2025) for detailed explanation on OOFS qualification or “calibration”. In other words, when the OOFS is upgraded, inter-comparing the old and new systems informs on the benefits of the upgrade and justifies “go/no-go” decisions.

• Adopt or refine technologies supporting large exchanges of information among the community: in this sense, the NetCDF file format and climate forecast standardization has greatly facilitated the “shareability” (Hernandez et al., 2015a, b) and prefigured the FAIR best practice (Findability, Accessibility, Interoperability and Reuse of digital assets), proposed more recently (Wilkinson et al., 2016).

An exceptionally illustrative intercomparison example emerged from the tragic crash of the Rio de Janeiro–Paris Air France plane in 2009 and the subsequent intensive search for the wreckage in the tropical Atlantic. Evaluation of the accuracy of current fields from OOFSs and observed products, a user-centric approach based on dispersion and Lagrangian metrics, was employed within an intercomparison framework. It was demonstrated that the ensemble mean yielded more reliable results compared to individual estimates (Drévillon et al., 2013). A similar approach was also adopted to identify the crash area for the March 2014 Malaysia Airlines flight MH370 in the Indian Ocean (Griffin and Oke, 2017; Durgadoo et al., 2021).

2.2 Intercomparison: key aspects to be addressed

Intercomparing routinely or during specific phases OOFSs and their products is now common practice in operational centres. However, various aspects need to be reiterated and addressed:

• Common validation/verification methodology needs to be adopted by all participants, preferably adopting recommendations, as reiterated in this report (Sotillo et al., 2025).

• Interoperability, shareability of products and common standards are key: the large number of products offered by the different centres cannot be spread in every single centre. The FAIR principles of best practice are essential.

• Representativity is a central aspect of intercomparison: scales and ocean processes represented in each product (observations and models) need to be correctly documented to reduce mis-interpretation when intercompared. In particular, the following points should be noted.

  • Re-gridding by downscaling or upscaling ocean products toward a common grid might generate errors and not conservative effects of ocean dynamics.

  • Comparing ocean re-gridded products with re-gridded observations containing different ocean scales might create double penalty scores.

  • Due to operational oceanography growing activity, it is worth remembering that an increasing number of products are available for each EOV, for each area. The Copernicus Marine Environment Monitoring Service (CMEMS) Data Store is a good illustration of this, with a large number of products derived from models or from space or in situ observations for a given EOV. This reinforces the importance of an a priori assessment of the representativity of each product before any intercomparison.

• Intercomparison is a first path toward ocean state estimation from various sources and products: it is promising to use novel approaches based on data mining, consensus clustering, machine learning, and other tools developed in the frame of ensemble estimation and forecast (e.g. Sonnewald et al., 2021).

• User-oriented metrics and process-oriented metrics are increasingly being implemented in operational centres. They are also new insight for establishing the performance of intercompared OOFSs into the user-oriented framework.

3 Ongoing ocean models and forecasting systems intercomparison activities

3.1 Class 4 metrics: model intercomparison in the observation space for verification forecast

Ocean observations provide an accurate estimation of the “ocean truth”. However, the Global Ocean Observing System (GOOS) provides a sparse representation over time of three-dimensional ocean dynamics. Their quantity and quality have increased substantially with permanent mooring and programmes such as Argo and the Global Drifter Program, together with satellite measurements (e.g. Tanhua et al., 2019). The GOOS is providing these recent years a valuable representation of the large-scale dynamics and some aliased representation of the ocean fine scales where measurements are performed. This led to the evaluation of OOFS performance by direct comparison with observations and to the definition of the Class 4 metrics detailed in Sotillo et al. (2025).

In summary, Class 4 metrics aims to compare observations with the equivalent model forecast at the same time and place, for different lead times (Hernandez et al., 2015a). Thus, these metrics, for different kinds of ocean variables, characterize the performance of a given OOFS against observations in the observation space. One of the advantages of using the observations as the reference frame is that other OOFSs can similarly be compared to the same data, in the same manner. Hence Class 4 metrics have been used since the beginning when comparing several OOFSs and their performance with the same “truth” (Hernandez et al., 2015a). When the observations are not assimilated by the OOFS, one can get a fully independent error assessment that can be statistically representative of the overall quality of the OOFS. Otherwise, one can consider that the overall error level is underestimated. However, this still provides an objective measure of the actual gap between the OOFS estimate and the “ocean truth” at the exact location and time of the observation used as reference.

Within GODAE OceanView, the Class 4 intercomparison project has been operating since 2013. A first set of intercomparison of six global OOFSs (Ryan et al., 2015) was an opportunity to present new metrics (radar plot, Taylor diagrams, best system mapping, bar charts, rank histograms, etc.). The same Class 4 information was also used with more specific metrics around Australia (Divakaran et al., 2015), with the objective of the Australian Bureau of Meteorology to identify a path of improvements for its own OOFS. This was also a first demonstration of one of the benefits of such intercomparison: the in-house routine validation in Australia took advantage of the internationally shared and compared multi-system Class 4 information to enhance its own daily basis verification procedures. The Class 4 intercomparison is still routinely performed (Fig. 1) and is continuously extended. A recent intercomparison based on Class 4 for surface velocity using drifters by Aijaz et al. (2023) offers an additional evaluation of OOFS surface dynamics performance, key for applications like search and rescue, marine pollution forecasts, and many other drift-dependent applications.

Figure 1 Operational centres and countries involved in a common intercomparison international framework during the last 20 years. Circles indicate their size and numbers the products/locations participating in the ORA-IP (Balmaseda et al., 2015). Green circles for ORA-IP only and red circles for centres that are contributing in addition to the Class 4 routine intercomparison (Hernandez et al., 2015a). Red stars indicate centres solely participating in the Class 4 intercomparison. Countries in pink, yellow and orange contribute, respectively, to Class 4, ORA-IP and both exercises.

Another issue of Class 4 comparison to observations was the routine evaluation of the overall quality of the GOOS. Performing comparisons with observations of several OOFSs also gives more confidence in identifying observation outliers and incorrect measurements: a feedback procedure was proposed to inform data centres that could carry out a second loop of data corrections, for the benefit of all data users (Hernandez et al., 2015b). This approach is now considered in the frame of the recent project SYNOBS endorsed by the United Nations Ocean Decade programme (Fujii et al., 2019, 2024). SYNOBS aims at evaluating the best combinations of ocean-observing platforms through observing system design carried out by different operational centres (e.g. Balmaseda et al., 2024a). The existing intercomparison framework will allow faster common assessment among the different contributors.

Mentioned above, comparison to observations raises the key issue of representativity, both from the observation and the modelling side. And subsequently, double penalty effects must be taken into account when measuring the skill of a given product for given scales or ocean regimes. It is necessary to carefully address the following questions: what are the scales sampled by a given observing system? What are the effective scales and ocean processes represented by a given OOFS? What ocean processes do they represent? The classical example is comparison of satellite altimetry and/or tide gauge observations with the sea surface height given by an OOFS: if the latter does not represent the tidal dynamics, obviously, observations need to be pre-processed to filter out tidal signals. This is the reason that the concept of “internal” metrics, aiming to measure the efficiency of the OOFS at the expected scales, was distinguished from the concept of “external” metrics, where operational products' reliability and fitness for purpose need to be assessed in the light of the user's requirements (Hernandez et al., 2018) and taken into account while performing intercomparisons. In addition, particular attention needs to be given to the representativity and uncertainty of observations. It is mandatory to take them into account while comparing several OOFSs with observations, in particular when referring to re-processed/re-gridded observation products (also called Level 4 or L4 type of observed products).

3.2 Ensemble forecast comparison: assessment through ensemble mean, ensemble spread and clusters

The atmospheric community developed ensemble forecasts, first to represent uncertainties of seasonal predictions considering the stochastic behaviour of atmospheric simulations. This was done using an individual forecasting system, by running a series of deterministic forecasts in parallel where some initial or forcing conditions were stochastically modified between members. The purpose of performing the intercomparison of the forecast members was to (1) identify common patterns from the probability distribution for eventually defining clusters, (2) compute probabilistic occurrences of specific events, and (3) use the ensemble spread as a proxy for forecast skill and performance assessment and try to separate outliers. The associated verification framework has been largely documented (e.g. Casati et al., 2008) and defined for the atmospheric components of the seasonal forecast activities (e.g. Coelho et al., 2019). For the ocean environment, this approach is currently used by weather prediction centres in charge of marine meteorology forecasting, i.e. wind and wave forecasts. For instance, the evaluation exercise performed by the National Oceanic and Atmospheric Administration (NOAA) National Centers for Environmental Prediction (NCEP), evaluating ensemble and deterministic forecasts, concluded, among other results, that the ensemble wave skill score at day 10 outperformed the deterministic one at day 7 (Campos et al., 2018). Another example is the recent intercomparison of seasonal ensemble forecasts from two centres contributing to the Copernicus Climate Change Service (C3S), which quantified their respective skill on sea surface height, ocean heat content and sea surface temperature (Balmaseda et al., 2024b).

At this stage, unlike weather prediction centres, ensemble forecasting from individual systems is not generalized in operational oceanography, although dedicated experiments are carried out in many areas (e.g. Pinardi et al., 2011; Schiller et al., 2020). And through specific data assimilation methods like the ensemble Kalman filter (Evensen, 2003), several centres are producing ensemble forecasts routinely (e.g. Lisæter et al., 2003; Keppenne et al., 2008; Seo et al., 2009). However, a large community effort dedicated to intercomparisons of ensemble forecasts produced by different centres has not yet been achieved.

Here we propose to illustrate ensemble approach benefits with a multi-system intercomparison as proposed by the CLIVAR/GSOP initiative (mentioned above) and the Ocean Reanalysis Intercomparison Project (ORA-IP) (detailed in Sect. 3.4 below and also discussed by Storto et al., (2019)). Figure 2 illustrates the assessment of a commonly used indicator for the so-called “Atlantic Niño” regimes in the tropical Atlantic, associated with the “Atlantic zonal mode” and targeting the equatorial cold tongue that develops in the Gulf of Guinea from April to July (Vallès-Casanova et al., 2020). All products –observation-derived-only and reanalysis estimates (see Balmaseda et al., 2015, for products' details) – give a consistent representation of the seasonal and interannual variability, from which an interannual trend can be deduced over the 1980–2024 period (ensemble-average trend in Fig. 2c of 0.02 °C per year). The ensemble average is computed like the multi-product-mean in Uotila et al. (2018) and without ARMOR3D, the observation-derived-only product used as “ground truth” (Guinehut et al., 2012), and without the GREP reanalysis, already an ensemble average of various reanalyses (Masina et al., 2015). Figure 2b shows the time series of the so-called SST (sea surface temperature) index: the box-averaged temperature anomalies relative to the annual climatology (computed with the ensemble average). All products exhibit the same interannual patterns, although some discrepancies are observed at intra-seasonal timescales. This is reflected by the small differences in the standard deviations computed for each time series over the denser period (1993–2023). A more precise view of the differences of each product's SST index with the ensemble average is given by Fig. 2a, quantified by their respective root-mean-square differences. Before 1993, the ensemble average is computed only with the ERA5 reanalysis and the OSTIA-observation-derived-only product, covering this period. Consequently, Fig. 2a exhibits a large discrepancy of these two products with respect to the ensemble average. The 1993–2023 period is chosen to assess the relative merit of each product, quantified using the ARMOR3D observation-derived-only product, not included in the ensemble-average computation in the Taylor diagram (Fig. 2d). First, one can see very large differences with OSTIA, the other observation-derived-only product, suggesting that the impact of their respective representativity of SST in the ATL3 box and possibly mapping/observation errors should be investigated further. The lesson here is that the “ground truth” also presents subjective drawbacks that need to be taken into account while measuring the relative merit in this multi-product ensemble assessment. The Taylor diagram reflects the very close performances of all products altogether in a cluster. The ensemble average performs better than individual reanalyses. The GREP multi-reanalyses product presents also good performances in representing the ATL3 index relatively to ARMOR3D. This confirms previous findings (e.g. Masina et al., 2015; Uotila et al., 2018; Storto et al., 2019) showing the “bias-reduction” benefits of ensemble averaging. In practice, the ensemble average provides a valuable estimate of the decadal SST trend in the ATL3 box. The ensemble-average estimate is also useful in identifying outliers.

Note that in recent methodologies, ensemble forecast comparison is performed using “ensemble clustering”, also called “consensus clustering”, which aims at producing a synthesis among an identified cluster from a given dataset (e.g. Hakobyan, 2010). The construction of the clusters from the initial dataset (here the different members of the ensemble forecast) can use many criteria. In the frame of GODAE OceanView, the Class 1 metrics were designed to compare OOFS variables on specific model grids and layers in similar ways (Hernandez et al., 2015b). In the Class 1 approach, OOFS outputs are re-gridded and resampled in a common grid and time frame (e.g. daily 2D model fields) and compared to a common reference (e.g. a regular L4 mapping of sea surface temperature from satellite retrievals). In this intercomparison, Class 1 files from various global OOFSs were used to compare and evaluate the quality of the ensemble mean; the weighted ensemble mean; and the k-mean clustering algorithm mean (Hartigan and Wong, 1979), which proved to be the more accurate (Hernandez et al., 2015b). Consensus clustering is now used for machine learning, and this might be one of the next stages associated with model products' intercomparison and ocean state estimation in the near future.

Figure 2 (a)–(c) Time series from 1980 to 2024 of SST products, monthly and spatially averaged into the ATL3 box located in the eastern equatorial band [20° W–0° E, 2.5° S–2.5° N] of the tropical Atlantic. (a) Differences relative to the ensemble average (root-mean-square differences (RMSDs) indicated in the legend). (b) The ATL3 index computed as anomalies relative to the climatology mean (standard deviations indicated in the legend). (c) The time series of box-averaged SST in the ATL3 box. (d) The associated Taylor diagram of the ATL3 index, using the ARMOR3D product as a reference. Statistics of root-mean-square differences, correlation with ARMOR3D and standard deviations for each product are given in the legend. All products were extracted from the Copernicus Marine Data Store and Climate Data Store.

3.3 Regional forecast intercomparison and nesting strategy evaluation

Over recent years, the validation methodology proposed by the GODAE global ocean community has been adopted by many operational regional centres (some examples are given by Hernandez et al., 2015b), in particular because the coastal community started to relate inside GODAE OceanView with the IVTT. Specific assessments also started to be carried out, like assessing the behaviour of the ocean under tropical cyclone conditions using several OOFSs and ad hoc metrics (Zhu et al., 2016) or predicting the beaching of Sargassum in the Caribbean using global and regional OOFSs (Cailleau et al., 2024).

On a regional basis, specific systematic multi-product validation tools are gradually developed (e.g. Lorente et al., 2016, 2019). These tools, operated by a given operational centre, are efficient essentially if an inter-operable data server policy is implemented among the operational ocean community, in order to allow the real-time intercomparison of different sources of products. In parallel, regional and coastal system evaluation relies on specific local observing systems, like high-frequency (HF) radar, offering an “ocean truth”, representing the ocean dynamics at higher resolution (Kourafalou et al., 2015), which cannot be represented by global OOFSs.

However, it is worth noting that comprehensive multi-product operational intercomparison is not common at regional scales. Unlike global OOFSs, there are rarely many fine-scale regional OOFSs that overlap in a given coastal region, even along the well-covered European marginal seas (Capet et al., 2020). And conducting a regional intercomparison gathering essentially global OOFSs would provide little information compared with the global intercomparison initiatives already underway.

But there is an increasing number of operational centres, or programmes like the CMEMS, that operate both regional and global systems over the same area and that have started to intercompare their different systems. For instance, two OOFSs of the same kind, Mercator and MFS (Mediterranean Forecasting System), are compared in the western Mediterranean basin, and their respective strengths and weaknesses are evaluated over specific subdomains (Juza et al., 2015). The benefit of improving the resolution of a regional OOFS is measured by comparing the coarse and fine grid systems using the same metrics (Crocker et al., 2020). In the CMEMS, most regional systems are nested into the global system. Hence, intercomparison between “parent” and “child” systems started to arise with the objective of measuring the benefit and added value for users of proposed regional and coastal products (De Mey et al., 2009). Several overlapping regional systems in the CMEMS can be compared to the global solution (Juza et al., 2016; Lorente et al., 2019). Examples can also be given for the Canadian Arctic and North Atlantic regional OOFS (Dupont et al., 2015), the US East Coast OOFS and reanalyses (Wilkin et al., 2022), and the Australian global and regional OOFS evaluations that focus on specific case studies and applications: disaster/search and rescue, defence/acoustic, and sea level/coastal management (Schiller et al., 2020). Some of these intercomparisons compare the regional OOFS of interest with several global products in order to measures both the local and global forecast skill, considering fine scales. In this case, using similar metrics, typically Class 4, for evaluating all these systems brings a series of questions. Which are the scales represented by the child system that is lacking in the parent system or in the observations? What is the impact of the different kind of forcings and different kind of assimilated dataset? How do errors propagate from the global to the nested system and degrade the expected seamless transition from the open ocean to coastal dynamics? How are specific ocean processes of interest represented in different systems? How reliable are they for end users' needs in different systems?

3.4 Evaluating retrospective views of the ocean dynamics: dedicated ocean reanalyses intercomparison project and ways to improve intercomparison methodologies

Past numerical simulations and ocean reanalyses were naturally the first step built in academia to study ocean processes over long periods, with the support of the increasing number of ocean observations over time and the improvement of assimilation techniques. Evaluation of such reanalysis representing decades of ocean behaviour through comprehensive intercomparison projects requires considerable resources and preparation. Most are conducted outside of routine operations by forecasting centres. They represent a milestone in progress in the field, from the point of view of both the evaluation of the system/reanalysis itself and the new validation methodologies that have been tested and implemented.

In the direct line of the GSOP project, the Ocean Reanalysis Intercomparison Project (ORA-IP) brought together more than 20 operational centres in order to intercompare more than 25 products (including observed products) spanning 20 to 50 years and focusing on eight EOVs – ocean heat content, steric height, sea level, surface heat fluxes, mixed layer depth, salinity, depth of the 20 °C isotherm and sea ice (Fig. 1). One of the objectives was to infer a new ocean state estimation of the global ocean, trying to reduce the so-called structural uncertainty, i.e. the uncertainty associated with the state estimation methodology, which cannot be sampled with a single system. Uncertainty is sensitive to the temporal variations of the observing system and to the errors of the ocean model, atmospheric fluxes and assimilation system, which are often flow-dependent and not easy to estimate. Following the Class 1 metrics approach, the ORA-IP is based on common grid re-interpolated products and monthly averages that were compared similarly over the 1993–2010 period under the responsibility of a leading expert for each of the eight EOVs. Results highlighted impacts of model resolution, components of the observing system assimilated, complexity of the ocean models and the data assimilation scheme, and quality of external forcing (Palmer et al., 2015; Shi et al., 2015; Storto et al., 2015; Toyoda et al., 2015a, b; Valdivieso et al., 2015; Chevallier et al., 2016).

New independent metrics were tested and used to evaluate each product and also the ensemble mean. The ensemble spread was identified as a measure of uncertainty. Following Storto et al. (2019), ocean reanalyses offer state-of-the-art representation of the past and present state of the global and regional oceans. Their accuracy depends on many factors, one of the most important being the observations available and the constraints they provide. Intercomparison helps in identifying the impact of their absence in the past and defines where they are most crucial in the quality of present and future reanalyses. And consequently, suggestions for improvements of the GOOS are provided.

Figure 2 shows that multi-product intercomparisons allow key indicator of the ocean environment changes to be inferred together with estimates of their uncertainties. Beyond reanalyses assessment based on EOVs, the next stage of ocean reanalyses intercomparison should first target key ocean processes that affect the climate system, identify their past occurrences, and better unravel their mechanisms and interactions, in order to estimate their present and future impacts. Machine learning approaches are expected to explore ocean variability in a multi-system framework more systematically and disentangle ocean key mechanisms for further identification in ocean simulations (e.g. Ahmad, 2019; Sonnewald et al., 2021; Salman, 2023). In particular, in ESM simulations, initial conditions are crucial: more realistic clusters of ocean reanalyses with better characterization of their errors and limitations (with or without the support of artificial intelligence) would ensure more reliable global and regional climate projections and associated skill assessment. Following this framework, ocean reanalyses intercomparison initiatives should also target end users' applications and societal impacts and identify requirements in terms of OOFS resolution, frequency and complexity, together with adequate observing systems, able to provide reliable and useful answers. Emerging international panels like the OceanPrediction Decade Collaborative Centre should help in providing intercomparison standards and recommendations from the user's point of view (Ciliberti et al., 2023). As already commented above, large and comprehensive multi-reanalyses intercomparisons are demanding and bring technical challenges in terms of storage, access, distribution and shareability. Cloud computing, ad hoc data mining technics and other artificial intelligence approaches will be needed to obtain valuable outcomes from the increasing number of available numerical ocean products resolving finer scales over longer periods.

3.5 A perspective of ocean reanalyses intercomparison: ocean state monitoring

An important outcome of the ORA-IP has been the development of the Real Time Multiple Ocean Reanalysis Intercomparison, carried out routinely every month by NOAA/NCEP, whose main objective is to gather operational hindcasts in order to perform ocean state monitoring (OSM) over the tropical Pacific, inferring the state of the ocean by computing the ensemble mean and identifying robust patterns using the ensemble spread (Xue et al., 2017). Note that OSM has growing importance in operational oceanography: through key EOVs it offers an assessment of the evolution of the ocean component as part of the real-time climate system evolution. Validation performed in the frame of OSM also provides a level of uncertainty for seasonal forecasts performed every month by many centres nowadays. OSM activity brought the CMEMS into routine calculation of Ocean Monitoring Indicators (OMIs), whose reliability and uncertainty are estimated through intercomparison of multiple products. Using OMIs, in 2018 the CMEMS started to produce the Ocean State Report (von Schuckmann et al., 2018) on an annual basis, now on its eighth edition (https://marine.copernicus.eu/access-data/ocean-state-report, last access: 29 January 2025).