Data Management

Data management plays an important role in the implementation of G3W. Appropriate data management should be established as a matter of priority to ensure that relevant and well-documented data, are easily accessible for both machines and humans through FAIR-compliant services, and that producing centres have access to the same sets of data to produce standardized outputs as well as participate in routine intercomparisons and to provide data access to users. It is expected that distributed data processing through regional hubs will increase the efficiency, quality, transparency, and timeliness of the data delivery. Processing should take care to provide information on measurement uncertainties and allow for generation of automatically quality-controlled, near real time data. Such data exchange ultimately will be provided through WIS 2.013 (planned to be operational in 2025) and therefore the integration of this data into these machine-ready data services is critical.

The (regional) communities involved in the existing operational networks have in recent years invested in moving to better, more FAIR and open data provision following the new WMO Unified Data Policy, under clear and standard open data licenses. All data processing and modelling should be open source and/or be properly documented as described in the action items below. Emphasis should be put on data provision for machine-to-machine operation, supporting documented automated workflows.

All observational data should be accompanied by a detailed and unbroken traceability chain and documented through rich descriptive and provenance metadata and the data should include quantitative uncertainty information as well as the usual data flags. These elements are needed to inform the users on the suitability of the data for different applications. All data and metadata flows should be integrated and feed into WIGOS.

The action items below build around four stages of data management from the instrument to data assimilation, models quality assurance and data provision to user community.

G3W Implementation Plan actions from Data Management block

Action D1: Data management life cycle stage 1; From raw instrument data to characterized product acceptable for international data exchange

The activities include identifying and surveying organizations for data exchange capabilities, identifying good practices and gaps in data processing, evaluating local data storage capabilities, enhancing data exchange from non-participating stations, developing training programs and open-source tools, and planning the transition of research activities into operational measurements. Success will be measured by periodic inspections of national and regional centers, data quality and contribution metrics, the publication of good practices, and the empowerment or establishment of global GHG data centers.

Action D2: Data management life cycle stage 2; Getting observational data from providers to operational centres for assimilation

The activities include defining the architecture and systems for automated, quality-controlled data transfer using FAIR principles, identifying targeted datasets, assessing WIS2.0 for data discovery and transport, defining data publication workflows, developing data policy and licensing recommendations, standardizing data and metadata content, integrating distributed networks and satellite data, and formulating data exchange requirements. Success will be measured by the efficiency and speed of the data transfer system and the availability of all required data to all centers.

Action D3: Data management life cycle stage 3; Making model output data from operational centres available for intercomparisons

The activities include developing recommendations for output data exchange architecture, assessing requirements for technical analysis and model intercomparisons, determining data field requirements, surveying existing data transfer methods, identifying suitable data formats, determining necessary statistical skills, and leveraging existing data storage mechanisms. Success will be measured by the reduction in labor and computational costs for model intercomparison activities.

Action D4: Data management life cycle stage 4; Model data discovery and distribution to end users

The activities include assessing user requirements, comparing G3W datasets against WIS metadata, identifying data and metadata standards, developing plans for metadata curation and API provisioning, establishing coordination mechanisms, creating training materials, developing a common landing page, and designing user satisfaction surveys. Success will be measured by the number of users, peer-reviewed publications, media presence, and the inclusion of data products in external discovery catalogues.

Action D5: Data repository for the prior emission and absorption fluxes

The activities include investigating requirements and solutions for data exchange and long-term storage of prior flux data, and securing a commitment from an infrastructure provider to develop and maintain the repository. Success will be measured by the estimation of data size, reduction in data transfer and access costs, and obtaining a commitment from the chosen infrastructure.

Action D6: Definition of prior data providers

The activities include establishing dataset requirements, defining data provider specifications and compliance procedures, setting Evaluation and Quality Control (EQC) criteria for model performance, and defining other Key Performance Indicators (KPI) criteria. Success will be measured by the development of a list of criteria for prior data providers and a set of tools and regulations for their assessment.

Action D7: Definition of the data policy for the repository of prior fluxes

The activities include defining the data license, access conditions, and user expectations for acknowledgement and citation. Success will be measured by the public availability of the data policy.