Skip to main content

Ensuring data quality

Data quality is a key aspect of testability because accurate, complete, and consistent data ensures that tests reflect real-world conditions. Poor data quality can mask defects, produce false results, or prevent meaningful validation of business rules. Ensuring high-quality data allows microflows and application logic to be reliably tested, making the system more predictable, maintainable, and robust.

To ensure that data quality is not compromised, every transaction that processes data must be strictly controlled. This is precisely where the principles of ACID come into play.

1. Introducing ACID: The Foundation of Data Integrity​

One of the most important paradigms for creating reliable software is the concept of the ACID transaction. An ACID transaction ensures that the database content remains valid and trustworthy, even when multiple objects are successively changed during a single business process. ACID is an acronym for the following four essential guarantees of data integrity:

  • Atomicity: the transaction is All or Nothing. Every step must succeed, or all changes are completely undone (rolled back). All mutated objects are treated as a single “unit”, called the scope.
  • Consistency: The transaction moves the data on the scope from one valid state to another valid state. All business rules and integrity constraints must be met before and after the transaction.
  • Isolation: Concurrent transactions should execute independently. The result of multiple simultaneous transactions must be the same as if they were executed one after the other.
  • Durability: Once a transaction is successfully committed, the changes are permanent and will survive system failures (power loss, crashes, etc.).

2. The ACID gap in Mendix​

Mendix provides a robust transactional environment which manages Isolation (I) and Durability (D):

  • Isolation (I): The Mendix Runtime wraps Microflow database actions in transactions and relies on the underlying database to manage concurrent access. The platform handles the complexity of locking and sequencing multiple simultaneous user requests.
  • Durability (D): The Mendix Cloud (or your target cloud deployment) ensures committed data is written to redundant, non-volatile storage. Once a transaction is finalized, the change is permanent and recoverable.

However, the platform does not enforce strict application-level Atomicity and Consistency. Key reasons why atomicity and consistency could not enforced by the platform are:

  1. Fragmented Commits: In Mendix, objects can be committed at any point within a microflow hierarchy. If multiple objects must be persisted as a single unit due to their dependencies, this flexibility can compromise business-level atomicity.

  2. Unenforced Consistency: There is no widely adopted or mandatory way how data validation must be applied before objects are committed to the database. While Mendix offers domain model validation and microflow checks, developers must explicitly and consistently implement them and adopt a framework to do so.

  3. Cross-Object Atomicity: It is not possible to commit a collection of objects of different types together with a single dedicated 'Commit' action. While a microflow action can commit a single object, ensuring that your coherent collection of multi-object changes (logical scope) are fully supporting atomic transactions requires careful manual error handling across the entire flow.

  4. Inconsistent Refresh Behavior: The mechanism to automatically update widget data, such as "implicit widget re-execution," is not guaranteed to trigger for every widget type. As a result, the state of an object displayed in the user interface can differ from its actual state in the Client Cache, often requiring developers to force manual refreshes to ensure accuracy.

The immediate result of this gap is that, despite the platform's reliability, the database can easily contain objects that are locally or relationally invalid, directly undermining data quality.

3. Solving the ACID problem​

The ACID gap give developers a challenge to implement and design Atomicity and Consistency themselves. There are two ACID properties that fall under the developer's control, providing the necessary foundation to write robust, testable, and high-quality applications by: