Building a more reliable test automation system

The evaluated scenarios typically took around 20 minutes or more. In consecutive executions, the old implementation showed a false-positive rate of roughly 60% or higher, even under normal conditions.

Recurring team complaints about instability and day-to-day friction prompted the investigation. The technical review found slow execution, sensitivity to page-structure changes and automation failures that still required manual investigation.

Presence and value validations were defined element by element. Expected product values were hard-coded even though the product shelf changed frequently.

I initiated the investigation and independently designed and implemented a new model for the original test scope.

The new Page Objects represented reusable page sections. They supplied a container and the field types supported within that area.

A reusable validator discovered fields inside the container, related them to their labels and compared their values with a map of expected results. Expected values could be maintained in text files instead of being embedded in the automation code.

• Page ObjectDefines the page area and supported field types.
• Data mapDefines each label and expected value.
• ValidatorDiscovers, extracts and compares values within the supplied scope.

The resilience model used condition-based explicit waits, controlled retries, container-relative location and a fresh element lookup after stale references.

I implemented a custom Cucumber reporting plugin that collected scenario duration, executed steps, screenshots, logs, errors and stack traces into an intermediate JSON document.

A configurable file-generation layer consumed that representation. Only the PDF generator was implemented. Screenshot timing was controlled through Cucumber hooks, allowing capture before or after each step.

The evidence did not automatically decide whether a failure belonged to the product or the automation. It provided a more structured starting point for manual investigation.

All approximately 16 scenarios that already had JUnit automation were migrated to the Cucumber-based model and integrated into the active test pipeline. They represented approximately 80% of the known backlog; the remaining items had not been automated in either approach.

Reusable form validations dropped from approximately five minutes to seconds in the evaluated scenarios. The evaluated scope also showed fewer maintenance tickets per scenario and broader automated functional coverage.

The model remained in use for its original pipeline scope. A proposal to extend it to other areas was not adopted because team members perceived the approach as too complex for everyday use.

At the time, I gave most of the presentation space to the internal architecture. Practical examples and representative before-and-after scenarios remained secondary. In retrospect, that may have made the intended usage and practical value harder to evaluate.

I may have focused on improving the internal architecture without making the adoption surface simple enough. A smaller public API, packaged libraries, representative examples and incremental demonstrations might have made the model easier to understand and assess.

Architecture has at least two audiences: the people who maintain its internals and the people who need to use it.

This case study passed confidentiality, rendered UX, accessibility and channel validation before publication.

Explore other evidence