|Met Office Hadley Centre observations datasets|
|> Home > HadIOD >|
None at this time
At present feedback format data are ony available to Met Office users. These webpages provide basic information for those interested in this dataset. Please contact us if you are interested in using these data.
NEMO NetCDF feedback format (hereafter ‘feedback format’) is a format used for model data assimilation (see NEMO-OCE pdf). A tool to extract data from the HadIOD database into feedback format was created as part of the ERA-CLIM2 project. Feedback format data from HadIOD.126.96.36.199 and HadIOD.188.8.131.52 were made available to ERA-CLIM2 participants and as a pre-release to Met Office colleagues. The HadIOD feedback format files have been through a few minor iterations following user feedback, each having its own version number (see 'Dataset versions' below).
Data are provided in daily files, the current version is v1.2 (see below). The v1.2 files presently cover 1978-2019 but could be extracted to cover other periods (HadIOD.184.108.40.206 covers 1850-present). For each day, two files are provided, one containing ocean profile temperature (in situ and potential temperature) and salinity data and the other containing sea surface temperature data. Each file contains the observed data along with metadata, bias corrections, measurement uncertainties and quality control decisions for each observation. The files contain several different bias correction and uncertainty variables; it is not possible to specify these for every observation and unspecified values are set to missing data. At a minimum we try to provide a quality flag and some estimate of uncertainty in each measurement (in some cases providing an uncertainty may not be possible). File contents and code tables used by certain variables are described in 'File contents' below. Sample file headers can be seen here:
Observed data are provided in 2D arrays (with stations on one axis and vertical levels on the other) and their station metadata are provided in 1D arrays. For profile data the vertical level dimension is 400 (i.e. a maximum of 400 vertical levels per profile) and for surface data the vertical level dimension is set to 1 (as these sample a single vertical level).
At present there is no user guide for the feedback format files. Basic information is provided on these webpages. We also strongly recommend reading the product user guide available for the HadIOD public file dataset which is of relevance here, in particular Section 2, which gives useful general information about HadIOD data and, importantly, describes the HadIOD observation error model and how to work with the bias corrections and uncertainties provided by HadIOD. The 'File Contents' section below has a table listing the bias correction and uncertainty variables in the feedback files and their public file equivalents, this will help interpret the public file product user guide for the feedback format files. The 'File Contents' section also has some further remarks comparing the two file types.
The HadIOD public file dataset includes supplementary files that provide additional sets of bias corrections to help users explore the bias correction uncertainty. The component of uncertainty explored by these corrections is increasingly important on longer space and time scales (e.g. if concerned with decadal trends). The use of these additional corrections is recommended and is described in the public file user guide. The supplementry files could be used with the feedback format files and are linkable via the DATABASE_IDENTIFIER variable.
Please contact us if you need help.
The feedback format file versions listed below are for data extracted from HadIOD.220.127.116.11. During development some preliminary feedback format data were extracted from HadIOD.18.104.22.168 with processing similar (but not identical) to v1.0 below.
Observations from several different sources have been ingested into the HadIOD.22.214.171.124 database. The following sources are used in the feedback format files (CMEMS was included in the sst files from v1.1 onwards):
ICOADS.2.5.1 - [Woodruff et al., 2011]. The primary source of surface ocean measurements in HadIOD. These data are included from 1850-present and provide sea surface temperature measurements (not salinity) to HadIOD.
Acknowledgement: Research Data Archive/Computational and Information Systems Laboratory/National Center for Atmospheric Research/University Corporation for Atmospheric Research, et al. 1984, updated monthly. International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Release 2.5, Individual Observations. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. https://doi.org/10.5065/D6H70CSV. Accessed 2011 onwards.
EN.4.2.0 / EN.4.2.1 - [Good et al., 2013]. The primary source of ocean profile data (and profile quality flags) in HadIOD. These data are included from 1900-present and provide temperature and salinity measurements on depth (also potential temperature) to HadIOD. From November 2017 the switch was made to using EN4.2.1 as EN4.2.0 was retired. From the perspective of HadIOD this switchover should be seamless with no discernible impact on the data.
Acknowledgement: EN.4.2.[0,1] data were obtained from https://www.metoffice.gov.uk/hadobs/en4/ and are © Crown Copyright, Met Office, 2020, provided under a Non-Commercial Government Licence http://www.nationalarchives.gov.uk/doc/non-commercial-government-licence/version/2/.
EN4 acknowledges the datasets contributing to its own construction (for full details see the EN4 webpages) and which HadIOD benefits from. These are:
E.U. Copernicus Marine Environmental Monitoring Service (CMEMS) drifting buoys - product INSITU_GLO_NRT_OBSERVATIONS_013_030 (Global Ocean In Situ Near Real Time Observations), http://marine.copernicus.eu/. Daily updates of global NRT observations from the main global networks from 2010-present provided by Coriolis (France). CMEMS drifting buoy sea surface temperature data are included in the HadIOD database from February 2015 to present. In the feedback files CMEMS is used as a replacement for ICOADS drifting buoy data from June 2016 onwards following a drop in the number of drifting buoy observations in ICOADS. CMEMS drifting buoy data are also included in the feedback files from 22nd-23rd October 2014 to fill a gap where there are no ICOADS.2.5.1 data.
Acknowledgement: HadIOD is generated using E.U. Copernicus Marine Service Information.
Met Office MetDB - in November 2016, surface buoy data (drifting and moored) were absent from the ICOADS Real Time updates. GTS buoy data from the Met Office Meteorological Database system were used in HadIOD as a substitute. CMEMS supersedes MetDB drifting buoys once available (sst files from v1.1 onwards).
HadIOD is subject to Crown copyright protection and is provided under a Non-Commerical Government Licence. Please do read this. Any onward licencing of the information must be for Non-Commercial purposes only.
When publishing work using the data, please quote the HadIOD version number (found in the files' 'source' attribute, see 'File contents' below), the feedback file version number (found in the files' 'file_version' attribute, see 'File contents' below), and cite the following:
Atkinson, C. P., N. A. Rayner, J. J. Kennedy and S. A. Good, 2014. An Integrated Database of Ocean Temperature and Salinity Observations, Journal of Geophysical Research: Oceans, 119, 7139-7163, doi:10.1002/2014JC010053
Users must also acknowledge the following:
If applicable, for traceability, please describe which HadIOD bias corrections were used. This information can be found in the file global attribute ‘source’. This is particularly important if any additional corrections from the supplementary files were used. See also 'how to cite the data' in the public file product user guide for more information and references. For example:
Please contact us if you need any assistance.
File contents are summarised in tables below.
A station (located horizontally and in time) may make one or more vertical profiles, with each profile comprising one or more observed levels (depths) of some geophysical variable. Variables listed below as 'station variables' contain station metadata (e.g. time or observation type) which apply to every observed level at a station. Variables listed below as 'data variables' contain the profile observations (temperature or salinity) and observation metadata (i.e. metadata which are potentially unique to each observation e.g. depth or a bias correction). The 'station variables' are 1D arrays with dimension N_OBS and the 'data variables' are 2D arrays with dimensions N_OBS x N_LEVELS. In the SST files N_LEVELS=1 and in the profile files N_LEVELS=400. A platform (e.g. a ship, buoy or profiling float) may make multiple stations; stations made by the same platform should share the same STATION_IDENTIFIER and STATION_TYPE (though in practice may not e.g. due to ID transcription errors).
Some general remarks on HadIOD Feedback files c.f. HadIOD public files:
See also the 'User Guide' section above.
Further useful information about some of the variables is given below.
'STATION_TYPE' variable values
This is the NEMO station type. For EN4 observations we use the EN4 WMO_INST_TYPE variable (see EN4 profile file format) which is a simplified subset of WMO code table 1770. For other source data we map to the categories below. These are only broad categories and some observations may not be in the correct category.
SST file QC variables
The Met Office Hadley Centre QC suite used for the ICOADS and CMEMS observations comprises basic sanity checks, position track check, climatology check and buddy check. Flags from the individual checks have been combined and mapped to the feedback format QC variables as listed below. If any of the individual checks fail, the corresponding feedback format QC flag is set to fail.
An updated version of the Met Office Hadley Centre QC suite (MDS4) has been used for the CMEMS observations. This performs a similar suite of checks to the previous suite (MDS3) used for the ICOADS observations but with improvements to the performance of checks such as the track, climatology and buddy checks. A repeated value check was also added.
The duplicate check used for ICOADS observations is created within the HadIOD suite. This flags near-surface temperature observations taken from ocean profiles that are non-preferred likely duplicates of data in the profile files. If using both the SST and profile feedback format files this flag should be applied. For CMEMS there is minimal expected duplication with the profile data and so a duplicate check is not applied.
The SST track check used for ICOADS (drifting buoy observations only) is an updated version of the drifting buoy QC checks described in Atkinson et al. . SSTs from individual drifters are tracked against a satellite-based reference and persistently poor quality observations are flagged. Position reports are used to flag buoys that have run aground or been picked-up. The coverage of the check is 1986-2015.
Feedback format flags use the following codes: 1=pass, 4=fail.
Profile file QC variables
The quality controlled profile observations in HadIOD are taken from EN4. The EN4 profile QC has many individual checks (see EN4 profile file format). In the EN4 data the overall outcomes of groups of individual checks are summarised in the POSITION_QC, PROFILE_POTM_QC, PROFILE_PSAL_QC, POTM_CORRECTED_QC and PSAL_CORRECTED_QC variables. These are mapped to equivalent variables in the profile feedback format files as shown below. The OBSERVATION_QC, POTM_QC and PSAL_QC variables apply to all levels in a profile. The POTM_LEVEL_QC and PSAL_LEVEL_QC variables are for individual levels in a profile.
Feedback format flags use the following codes: 1=pass, 4=fail.
Bias correction and uncertainty variables
Bias corrections should be added to the corresponding observations, e.g.:
Further rules on applying the correctons may be given in the files' 'usage' global attribute. Do read this. For example, the 'usage' may state that "DEPTH_CORR and SST_OBS_TYPE_CORR must be applied together (and not independently)" i.e. if applying SST_OBS_TYPE_CORR to SST_OBS, then DEPTH_CORR must also be applied to DEPTH (this might be because, for example, SST corrections are made relative to some reference level).
The feedback format files use the same HadIOD observation error model as the HadIOD public files. The public file product user guide explains the HadIOD error model and how to work with the bias corrections and uncertainties provided by HadIOD. Reading Section 2 of the user guide is strongly recommended. The table below lists the equivalent bias correction and uncertainty variables in the feedback format and public files, this will help interpret the public file product user guide for the feedback format files.
Note that the profile feedback format files do not include bias correction and uncertainty variables for quantities that are not currently specified for profile data (shown as N/A).
Commercial and media enquiries
You can access the Met Office Customer Centre, any time of the day or night by phone, fax or e-mail. Trained staff will help you find the information or products that are right for you.
© Crown Copyright