DataStructure

Structure

The overall structure of the data-flow at the web-based platform Maon can be seen in the following figure.

Main data-flow of the integrated energy market data storing, modelling and analysis environment
Figure 0: Main data-flow of the integrated energy market data storing, modelling and analysis environment

The multi-user, multi-interface and multi-format data entry and extraction process benefits the streamline collecting and distributing data. Maon can be used at the same time for multiple market studies with varying data formats like data with aggregation on different levels. Maon is designed to be as robust as possible and to be accessible at any time anywhere. The model data available in an comma-separated values (CSV) file offers the possibility to apply directly Bash, VBA or Python scripts on the input and output files reducing manual data processing. Further, different data sources and data sinks can read and write en masse in parallel from different locations into and from the database via the public web application programming interface (API).

The import procedure provides the user a tool to collect, check and evaluate flexibly the data required for modelling (see import). The database allows highly parallelized data loading and unloading processes, which can be used to capture a large mass of data within a short time from different sources at the same time. Near real-time data capturing is possible.

The Maon database uses a base model scheme (see input and output) as well as an extensive metadata management (see metadata). The metadata management enables integrations of Maon in other advanced data processes through data-sided links and dynamically configurable scheme-free metadata as necessary by the user. So the reuse and traceability to any other data scheme can be achieved. Thus, an agile and quickly adaptable data model for changing business requirements can be provided. Maon can capture free-definable new values and columns in the metadata management. It allows users to adapt the Maon database on its own data processing and collections needs through an flexible way of data structuring. At the same time the database follows the approach that import queries ensure compliance of the data model to keep the data quality high.

The database holds automatically non-editable input data snapshots of every check and run (see check) to permanently enable users to restore previous states, to re-run simulations, to have backups in the event of deleted data locally at the working station of the user and to keep track of any input and metadata changes. This way Maon ensures full transparency and traceability of model data and metadata. Multiple data quality gates not only process data in error-cases with concrete descriptions, but also display error lists and provide automatic procedures to make it acceptable via, e.g., data corrections to reach the most-efficient inclusion of data sources into the database.

The optimization-based electricity market model can handle Europe-wide input data for multiple years in hourly granularity (see optimization). It simulates degrees of freedom in exchange, demand, renewables, batteries, thermal power plants and hydropower cascades as a deterministic or stochastic Monte Carlo simulation run. The input data is processed by a three-staged optimization problem solving procedure (see procedure). Preliminary results can be derived via comprehensive aggregation methods in just some minutes (see fast mode). This way Maon is the fastest electricity market simulation available.

Raw results are automatically processed to different result matrices and can be exported through for example formatted CSV files. Results like power prices, schedules, exchanges, outages, revisions, social welfare measures or component-wise cost-benefit measurements can be visualized and compared after runs instantly in the graphical user interface (see result analysis).

Quality

The quality of insights derived via Maon depends not only on the model performance, but also on the input data quality. Therefore, a detailed and effective validation of the input data is of the highest priority. Maon validates automatically every input data character, format, link and much more before starting simulations and reports users the inconsistencies via error lists and direct solving proposals (see check). In addition, the Maon database accepts only validated input data and reports back also error lists and solving proposals. It is technically impossible to derive inconsistent database entries due to human errors or user-sided faulty implementations.

History

The Maon database provides a historization by design to enable features like old data state restorations and reportings. The input data state is logged via data checks and runs. Every data check and run folder comprises its creation time stamp and the full input data that cannot be changed. This snapshot folder can be used to restore the according database state for a given time stamp. This input data is fed by the project and scenario data that can be changed at any time and states the single source of truth. To prevent accidental loss of data, an automated versioning is included and simulations can be locked to prevent accidental deletion. To support tracing input database changes, dates of editing and involved users are recorded.

Metadata

In the context of changing and new metadata that needs to be added, Maon provides optional non-limited columns and rows for appropriate persistent storing. Through the metadata storing feature, productive data and metadata can be traced closely related to model data. Freely defined metadata can be used, for example for:

  • Component look-ups and matchings
  • Integrations into processes
  • Reportings
  • Data audits
  • Data source references
  • Data target locations
  • Input data accuracy estimates
  • Creation dates
  • Raw input data
  • States during data collections
  • Mappings of connections between entities, properties or times

Metadata needs only to be linked to any entity (generator, storage, consumer, grid or virtual disabled component) of the model data to enable the data access. The metadata structure can generate a very high number of rows compared to the model data. It provides efficiency and flexibility in the data loading process as the model can be used to map rows, columns and mutations within the database without having to adapt the base model scheme. Due to this metadata management, flexible adjustments to the data contents are possible without hindering or disrupting existing model data or connected processes by creating new metadata matrices or link or create new links to existing metadata. Metadata can have any data scheme to ensure a maximum flexible and fast data work flow. The metadata management makes it possible to react flexible to changed input data (for example new columns for a data category) due to the non-existent need of introducing further setups while still maintaining the full historization, transparency and model features.

Access

Maon is accessible via a web interface (“clientless solution”). Data transferred and stored within the database is therefore considered highly sensitive. Only authorized users can access the data. The user administration is currently carried out by the Maon-Team available through the help desk.

All files are securely transferred with access control. Access to the platform is possible for authorized users through the graphical user interface login and the web service interface login for Maon embedding in third-party software. For efficient integration into other solutions, data queries can be applied via the Maon API, so that users can transfer data from Maon into their solutions or provide data from their solution to Maon. It is not possible to enter micro services behind the API by users directly due to security reasons. All internal data transfers behind the API use leading-edge encryption methods.

JSON Web Tokens (JWT) handle data and service access controls. Such tokens are encrypted with Rivest–Shamir–Adleman (RSA) (1024 bit, at least weekly key rotation) and hashed to the user signature. Maon API services know the private key internally and can therefore verify user signatures. This verification method ensures that Maon has issued the token and that no manipulation was done. Data transfer is based on the Transport Layer Security (TLS) encryption in Hypertext Transfer Protocol Secure (HTTPS). This encryption method guarantees that data cannot be interpreted between Maon servers and user browsers. See also the privacy policy.