Exstream Regression Testing Tool (composition & comparison)
Exhaustive test case generation and automated output comparison
Context
In a finance/credit environment, document evolutions on Exstream statements needed to be secured. Manual QA testing was costly and incomplete given the combinatorics of discriminants (variants, composition rules, brands/products). The goal was to ensure non-regression from one version to another by automating both exhaustive test case generation and output comparison.
Objectives
- Build an automated regression testing tool between two Exstream versions
- Exhaustively generate test cases from document engineering settings (discriminants) and a stream model
- Automatically compose multi-format outputs (AFP/PDF/HTML) for each version from the same test stream
- Batch compare outputs and produce a report of differences only for quick validation
Approach
Development of a Perl module reading all document engineering settings to identify, through discriminants, the complete list of test cases covering all statement template variants. Generation of a test data stream from a reference model: instantiation of all occurrences by systematically substituting discriminant values and identification data (e.g. customer name replaced by test case identifier) to make each output traceable. Execution of Exstream composition in batch mode for a first version (baseline) and production of AFP/PDF/HTML outputs. Reuse of the same stream with the second version (containing evolutions) to generate the second set of outputs. Implementation of automated comparison processing: either by comparing HTML via a dedicated Perl script, or via HP Output Compare in batch to list discrepancies and produce a PDF of differences, quickly consultable to validate that observed changes correspond only to expected evolutions. Testing, delivery and documentation writing.
Deliverables
- Complete Perl tool for test case generation and exhaustive data stream
- Batch Exstream composition chain multi-format (AFP/PDF/HTML) on two versions
- Automated comparison processing (HTML compare and/or HP Output Compare batch) with PDF difference report
- Documentation + operations guide, testing and delivery
Results achieved
- Exhaustive variant coverage through automatic test case generation
- Drastic reduction in QA testing time and reliable regression testing
- Comparison focused on differences via a readable and quick-to-validate PDF report
- Industrialized and reproducible process for each document evolution
