Archive by Author | stevenhubbard2014

Integration of CHARME with the Copernicus CQC Service

Introduction

Airbus Defence and Space operates the Copernicus Coordinated Quality Control (CQC) service (http://gmesdata.esa.int/web/gsc/CQC) on behalf of the European Space Agency (ESA). This is in support of the space component of the European Union (EU) Copernicus programme, which provides the Copernicus Service Projects (CSPs) with access to datasets from a range of contributing satellite Earth Observation missions.

The Copernicus CQC service is concerned with the monitoring and storage of all quality reports related to the missions, datasets and products involved in the Copernicus programme.

The provision of EO data to the Copernicus services by the Copernicus Space Component Data Access (CSCDA), now in its third phase of operations, is via a set of datasets defined in a Data Access Portfolio (DAP). The assessment of the quality of the DAP datasets and of the contributing missions is the responsibility of the Copernicus Coordinated Quality Control (CQC) service.

CQC Service Tasks

While the data quality of each delivered dataset remains the responsibility of the contributing mission, the CQC role is to perform further independent quality analysis, respond to and coordinate anomaly investigations and provide harmonisation and traceability of the quality information.

The service provided by CQC is divided into 10 separate tasks. Those related to the core quality control work are highlighted in Figure 1 in terms of two viewpoints:

Figure 1 - CQC Service Tasks

Figure 1: CQC Service Tasks

The analysis of representative data products (Task 3), handling of user feedback (Task 5) and generation of quality control synthesis reports when a dataset is closed (Task 6) are all related to Copernicus Datasets. However, the support of Copernicus Contributing Mission (CCM) integration (Task 9), CCM harmonisation (Task 7), sample data analysis (Task 4) and user feedback (Task 5) all provide information relevant to the CCM perspective.

A series of Python tools have been developed to support the automation of the image quality checks, which produce structured information that is stored and then viewed in an Oracle database.

However, the image quality checking process also yields unstructured, ad-hoc information, which might reveal patterns and trends that need to be addressed. This CQC commentary metadata can now be processed using the Open Annotations linked data model developed in the FP7 CHARMe project. To this end a new dedicated CHARMe server has been setup and configured for the CQC service.

CHARMe Adaptions for CQC

For deployment in the CQC, the standard Web page plug-in has been replaced with a custom client, written in Python. The basic Open Annotation model has also been extended to support multiple targets (see Figure 2). Specifically, each commentary body is annotated to two targets:

  • the associated Data Access Portfolio dataset – to support the generation of synthesis reports when a Dataset is closed;
  • the source Copernicus Contributing Mission – to facilitate the collation of quality issues and detection of patterns.
Figure 2 - Multiple Target Annotation Model for CQC Commentary

Figure 2: Multiple Target Annotation Model for CQC Commentary

For the CQC, two virtual machines have been configured: one hosts a deployment of the CQC CHARMe node and the other hosts a Web server. Figure 3 illustrates the organisation of the Web server which provides the URLs for both the annotation targets and bodies:

Figure 3 - CQC CHARMe Web Server

Figure 3: CQC CHARMe Web Server

The index for the Web pages links to two separate webpages, one hosting external web links to Copernicus Contributing Mission websites and the other links to internally held Copernicus Dataset definition pages. Both of these webpages also embed the CHARMe JavaScript plugin. The Web server additionally serves the CHARMe plugin itself and the CQC image analysis reports containing the ad hoc commentary metadata about detected anomalies. These reports, along with the URL links to background documents and user-complaint investigations, provide the bodies of the CQC commentary annotations.

At present the CQC CHARMe node operates on an internal network within Airbus DS. However, as the commentary metadata is added the information resource should become increasingly useful to both data providers and CCMEs. Therefore, a future evolution might be to allow external access to the CQC CHARMe node.