Structure
The overall structure of the data flow at the web-based platform Maon can be seen in the following figure.
The multi-user, multi-interface and multi-format data entry and extraction process enables the streamlined collection and distribution of data. Maon can be used simultaneously for multiple market studies with varying data formats like data with aggregation on different levels. Maon is designed to be as robust as possible and to be accessible at any time anywhere. The model data available in a comma-separated values (CSV) file offers the possibility to apply directly Bash, VBA or Python scripts on the input and output files reducing manual data processing. Further, different data sources and data sinks can read and write en masse in parallel from different locations into and from the database via the public web application programming interface (API).
The import procedure provides the user with a tool to collect, flexibly check and evaluate the data required for modelling (see input import). The database allows highly parallelized data loading and unloading processes, which can be used to capture a large volume of data within a short time from different sources at the same time. Near real-time data capturing is possible.
The Maon database uses a base model scheme (see input and output) as well as an extensive metadata management (see metadata). The metadata management enables the integration of Maon in other advanced data processes through data-links and dynamically configurable schema-free metadata as necessary by the user. So the reuse and traceability to any other data scheme can be achieved. Thus, an agile and quickly adaptable data model for changing business requirements can be provided. Maon can capture user-definable new values and columns in the metadata management. It allows users to adapt the Maon database to its own data processing and collection needs through a flexible way of data structuring. At the same time the database follows the approach that import queries ensure compliance of the data model to keep the data quality high.
The database holds automatically non-editable snapshots of input data for every check and run (see data check) to permanently enable users to restore previous states, to rerun simulations and maintain backups in the event of deleted data locally at the working station of the user and to keep track of any input and metadata changes. This way Maon ensures full transparency and traceability of model data and metadata. Multiple data quality gates not only process data in error cases with concrete descriptions, but also display error lists and provide automatic procedures to make it acceptable via, e.g., data corrections to reach the most-efficient inclusion of data sources into the database.
The optimization-based electricity market model can handle Europe-wide input data for multiple years in hourly granularity (see optimization). It simulates degrees of freedom in exchange, demand, renewables, batteries, thermal power plants and hydropower cascades as a deterministic or stochastic Monte Carlo simulation run. The input data is processed by a three-staged optimization problem solving procedure (see procedure). Preliminary results can be derived via comprehensive aggregation methods in just a few minutes (see fast mode). This way Maon is the fastest electricity market simulation available.
Raw results are automatically processed to different result matrices and can be exported through, for example, formatted CSV files. Results like power prices, schedules, exchanges, outages, revisions, social welfare measures or component-wise cost-benefit measurements can be visualized and compared after runs instantly in the graphical user interface (see result analysis).
Quality
The quality of insights derived via Maon depends not only on the model performance, but also on the input data quality. Therefore, a detailed and effective validation of the input data is of the highest priority. Maon validates automatically every input data character, format, link and much more before starting simulations and reports inconsistencies to users via error lists and direct solving proposals (see data check). In addition, the Maon database accepts only validated input data and reports back also error lists and solving proposals. It is technically impossible to have inconsistent database entries due to human errors or user-side faulty implementations.
History
The Maon database provides versioning by design to enable features like old data state restorations and reportings. The input data state is logged via data checks and runs. Every data check and run folder contains its creation timestamp and the full input data that cannot be changed on the cloud platform. This snapshot folder can be used to restore the corresponding database state for a given time stamp. This input data is fed by the project and scenario data that can be changed at any time and represents the single source of truth. To prevent accidental loss of data, automated versioning is included, and simulations can be locked to prevent accidental deletion. To support tracing input database changes, dates of editing and involved users are recorded.
Metadata
In the context of changing and new metadata that needs to be added, Maon provides optional unlimited columns and rows for appropriate persistent storing. Through the metadata storing feature, productive data and metadata can be traced closely related to model data. Freely defined metadata can be used, for example for:
- Component look-ups and matchings
- Integrations into processes
- Reportings
- Data audits
- Data source references
- Data target locations
- Input data accuracy estimates
- Creation dates
- Raw input data
- States during data collections
- Mappings of connections between entities, properties or times
Metadata only needs to be linked to any entity (generator, storage, consumer, grid or virtual disabled component) of the model data to enable the data access. The metadata structure can generate a very high number of rows compared to the model data. It provides efficiency and flexibility in the data loading process as the model can be used to map rows, columns and mutations within the database without having to adapt the base model scheme. Due to this metadata management, flexible adjustments to the data contents are possible without hindering or disrupting existing model data or connected processes by creating new metadata matrices or links to existing metadata. Metadata can have any data scheme to ensure a maximum flexibility and fast data workflow. The metadata management makes it possible to react flexibly to changed input data (for example new columns for a data category) due to no need to introduce further setups while maintaining full versioning, transparency, and model features.
Access
Maon is accessible via a web interface (“clientless solution”). Data transferred and stored within the database is therefore considered highly sensitive. Only authorized users can access the data. The user administration is currently handled by the Maon-Team available through the help desk.
All files are securely transferred under access control. Access to the platform is possible for authorized users through the graphical user interface login and the web service interface login for Maon embedding in third-party software. For efficient integration into other solutions, data queries can be applied via the Maon API, so that users can transfer data from Maon into their solutions or provide data from their solution to Maon. Users cannot directly access microservices behind the API for security reasons. All internal data transfers behind the API use leading-edge encryption methods.
JWT handle data and service access controls. Such tokens are encrypted with RSA (1024 bit, with at least weekly key rotation) and hashed to the user’s signature. Maon API services know the hashed private key internally and can therefore verify user signatures. This verification method ensures that Maon has issued the token and that no manipulation was done. Data transfer uses TLS encryption over HTTPS. This encryption method guarantees that data cannot be interpreted between Maon servers and user browsers. See also our privacy policy.