Chapter 7 Monitoring, Evaluation and Adaptation

After the successful implementation of the spatial data infrastructure (SDI), regular Monitoring, Evaluation and Adaptation should be carried out on a regular basis, as these are essential processes to confirm the SDI’s effectiveness and relevance over time. Continuous monitoring involves tracking key performance indicators (KPIs) such as data accuracy, user accessibility, and system interoperability to identify areas needing improvement. Regular evaluation, through stakeholder feedback and performance reviews, assesses the SDI’s impact on marine planning and its alignment with environmental and regulatory goals.

Adaptation is the process of making iterative adjustments based on these evaluations, incorporating technological advancements, evolving regulatory requirements and changing user needs. This dynamic approach verifies that the SDI remains a robust and responsive tool.

7.1 Methods for SDI Monitoring and Evaluation

Evaluating an SDI is a process where a variety of methods can be applied to capture all relevant aspects. These methods include:

  • Stakeholder feedback through surveys, questionnaires, interviews and focussed discussion groups: Collect quantitative and qualitative feedback from stakeholders on usability, performance, satisfaction, and other aspects. Group discussions can give interactive insights into their experiences, expectations and suggestions.

  • Practical user feedback – usability testing: Direct observation and assessment of how users interact with the SDI to identify usability issues and areas for improvement.

  • SDI performance analysis: Measurement of technical SDI performance indicators such as response time, uptime, data throughput74 and error rates to evaluate efficiency and reliability.

  • Data quality assessment: Evaluation of the accuracy, completeness, consistency and timeliness of the data within the SDI.

  • Security audits: Comprehensive security evaluations to identify vulnerabilities and assess the SDI’s ability to keep data protected.

  • Compliance and standards review: Assessment if the SDI meets relevant industry standards and regulations for data management and security.

  • Scalability testing: Test the SDI’s ability to handle increasing amounts of data and user load without decreases in performance.

  • Backup and recovery testing: Assess the backup and recovery processes in case of system failure.

  • Cost-benefit analysis: Analyse the financial aspects, including implementation and maintenance costs, against the benefits and value gained from the SDI.

7.2 Establishing success criteria for the SDI

Establishing success criteria for the SDI involves setting clear, measurable goals to verify the infrastructure effectively supports the vision, goals and objectives (see Chapter 2 Task 4) of the MSP initiative.

7.2.1 Key Performance Indicators (KPIs)

Key performance indicators (KPIs) are measurable values that measure how effectively the SDI is achieving its key objectives. They provide a focus for strategic and operational improvement, helping to understand the SDI’s performance and identify areas needing attention. KPIs can be measured through a combination of the SDI evaluation methods outlined in 7.3.

Choosing the right KPIs relies upon a solid understanding of what is important to the SDI in terms of supporting the MSP initiative. What is considered important often depends on the point of view of particular components: KPIs useful for achieving marine planning goals are often different from the KPIs relating to database performance. In that context, KPIs can be defined as the repeated achievements of some operational goals (e.g. high efficiency of SDI operations, high user satisfaction), or in terms of progressing towards particular strategic goals (e.g. adding a new functionality to the web portal).

KPIs for SDI in the MSP context may include the following metrics (Figure 35):

  • User experience: User experience refers to how efficiently users can access, interact with, and derive value from the SDI. This includes 2 components:

    • Meeting users’ data needs: An effective SDI must provide comprehensive datasets covering various themes such as marine ecosystems, shipping routes, fishing zones, coastal development, etc. Having diverse and extensive datasets available helps meet the diverse needs of different user groups (see Chapter 3.1.1). User-centric design principles makes sure that SDI interfaces and tools are intuitive and easy to use (see Chapters 3.2.5 and 4.4). Additionally, SDIs should offer tools that allow users to customise the data and services according to their specific needs, such as capabilities for data filtering, thematic mapping, and scale adjustments. Please see Chapter 2 Task 3 for details about user needs assessment.

    • Accessibility: SDI should be made available through user-friendly web services and APIs, ensuring that even users with limited technical expertise can access and utilise the data. User-friendly interfaces and tools are essential for SDI accessibility. Emphasising usability testing and user training enhances the overall user experience and assures that tools are accessible to a wide audience. Additionally, user support should involve speedy acknowledgement and solutions for reported issues. Please see Chapter 4.4 for web portal accessibility in terms of ease and speed of user access, data visualisation and data download.

  • Data quality: Measures the accuracy, completeness, and reliability of data contained in the SDI. High data quality implies that spatial data accurately represents real-world features, is consistently updated and complies with most recent standards. It also verifies that metadata is properly documented and accessible. The EU INSPIRE Directive, for instance, provides detailed guidelines on data specifications and metadata standards to support high-quality and consistent data management. Please see Chapter 3.3 for details about good practices for data.

  • Interoperability: Assesses the SDI’s ability to integrate and exchange data seamlessly with other systems and datasets across different platforms and stakeholders. Ensuring interoperability is crucial for SDI. This involves adopting international standards like those from the Open Geospatial Consortium (OGC) for data formats, web services, and interfaces. SDI promotes collaboration by creating networks of data providers and users. This is evident in initiatives like the UN-GGIM, which facilitates international cooperation and data sharing to address global challenges. The ability to integrate SDI with other information systems, such as Geographic Information Systems (GIS), Computer-Aided Design (CAD) systems, and remote sensing platforms, enhances its utility and reach. Please see Chapter 2 Task 8, Chapter 3.2.4 and Chapter 4.1 for more details about SDI interoperability.

  • SDI performance: Evaluates the efficiency and responsiveness of the SDI in processing and delivering data. It encompasses factors such as data retrieval times, system uptime, and the ability to handle large datasets and complex queries. High system performance facilitates that users can access and analyse spatial information quickly and reliably, and that data can be shared with other systems without delays. Please see Chapter 3.2 for more details about data architecture in terms of data management, storage and maintenance, and Chapter 4.2 for efficient handling of large data volumes.

  • SDI service availability and reliability: Measures the percentage of time the SDI is operational and accessible to users without failures or errors. High scores confirm that users can reliably access and use spatial data whenever needed. Please see Chapter 3.2 and Chapter 4 for more details about optimizing data and web architecture.

  • Security and compliance: Evaluates how well the SDI protects sensitive spatial data and adheres to regulatory requirements. Security measures include safeguarding data from unauthorised access, breaches, and cyber threats. Compliance involves meeting legal, regulatory and industry standards related to data handling and privacy (i.e., data protection laws and regulations). Please see Chapter 5 for more details about security and privacy.

Figure 35: Key Performance Indicators (KPIs) for evaluating an SDI.

© UNESCO

7.2.2 Service Level Indicators, Objectives and Agreements

Within the KPI framework, Service Level Indicators (SLIs), Service Level Objectives (SLOs) and Service Level Agreements (SLAs) allow for a more detailed, structured approach to defining, measuring and formalising the quality of service. These metrics are in close alignment with the KPIs to track and improve the SDI performance.

SLIs are specific metrics used to measure the performance of a service, focusing on aspects such as availability, uptime, response time or throughput. These indicators help quantify the quality of service provided and form the basis for setting SLOs. SLOs set the target values or thresholds for these SLIs, defining what acceptable performance looks like. SLAs formalise these expectations in a contract between service providers and clients that outline the SLOs and the consequences if these targets are not met. Table 10 gives examples of what SLIs, SLOs and SLAs may include in the case of evaluation an SDI:

Table 10: KPIs with associated examples of SLIs, SLOs and SLAs for SDI evaluation. Note: All percentages included in the table should be seen as indicators and not fixed values to be achieved for specific SDIs; the individual target values for SLOs depend on the SDI project and its context. (Source: Elaborated by authors with input from multiple experts and bibliographic reference).

Metrics (KPI) SLI SLO SLA
User experience: meeting user data needs Comparison of available datasets within the SDI against the data requirements specified by users or stakeholders. X% of data needs specified by users must be delivered by the SDI. Definition of user needs to be met in terms of data availability.
User experience: accessibility

Evaluation of:

- The intuitiveness of the user interface.

- The availability of user support.

- The system’s capacity to handle concurrent users.

- The SDI interface is scored as highly intuitive by users.

- The SDI’s user interface is available 24/7, with uptime of at least 99.5%

- Support requests are responded to within 24 hours.

-The system is capable of handling several concurrent users.

Specification of the visual appearance and components to be included in the user interface.

-Specification of the support level provided, including response times for user queries and issues, as well as the procedures for reporting and resolving problems.

For example: support requests will be acknowledged within 1 business day and resolved within a set timeframe, such as 48 hours.
Data quality Measurement of data accuracy, completeness and timeliness, and metadata documentation and accessibility. 95% of the data and metadata in the SDI must meet current accuracy and completeness standards.

-Definition of regular data and metadata updates.

-Definition of data and metadata quality benchmarks.
Int eroperability Assessment of the SDI’s ability to integrate and exchange data seamlessly with other systems and datasets. Achieve 99% successful data integration across all supported formats and systems. Specifies what data should be exchanged with other platforms, and which platforms these are.
SDI performance

-Measurement of response times for data queries.

-Measurement of the speed of data processing and visualisation.

Set a target for query response times, for example:

- 90% of data queries are processed within 2 seconds.

- Map rendering times to be under 5 seconds for complex v isualisations.

Definition of acceptable performance levels for the SDI, including response times for data queries and system processing speeds. For example:

-90% of queries should be completed within 2 seconds.

-complex map visualisations should load within 5 seconds.
SDI service availability reliability Tracking of the uptime and availability of the SDI: monitoring downtime, maintenance periods, and overall system stability.

- System uptime of 99.9%.

- Any scheduled maintenance communicated in advance.

- Not exceeding a specified downtime limit, such as 4 hours per year.

- Specification of the expected SDI uptime, often aiming for high availability targets such as 99.9% uptime.

- Details of any planned maintenance windows and notification procedures for unscheduled outages.
Security & compliance Evaluate if the SDI meets necessary security standards and regulatory requirements.

-100% of data transactions are encrypted.

- The SDI meets all relevant regulatory requirements, including regular security audits and compliance reports.

 Definition of the security measures and compliance requirements that the SDI must meet, such as data encryption standards, access controls, and adherence to relevant regulations and standards.

7.3 Evaluating and improving the SDI

After establishing the KPIs as well as SLIs, SLOs and SLAs, the evaluation and improvement of the SDI can be carried out with a structured approach. By regularly reviewing performance against these metrics, areas of weakness within the SDI can be identified, which can then drive optimization efforts and adjustments of configurations or scaling of resources to meet evolving demands. Figure 36 shows the suggested structure of an SDI evaluation process, with each evaluation step focussing on distinct KPIs and linking back to specific SDI aspects for improvement in case the target performances have not been achieved.

Figure 36: Suggested decision diagram for the SDI evaluation process. Each step focuses on distinct KPIs and provides links back to specific chapters of this Data Toolbox for improvements in case target performances have not been met.

©UNESCO

  1. Data throughput refers to the amount of data that can be transmitted, processed or accessed within a given time, usually measured in units like bits, bytes or megabytes per second. It reflects how efficiently the SDI handles data flow between servers, services and users.↩︎