Amnis Dataflow Validator
The purpose of the Amnis Dataflow Validator is the validation of CSV or columnar data. In Dataflow Validator, data structure and logic definitions are described in the metadata files. Input files are validated according to this configurable file. The Dataflow Validator can be run either as an standalone executable, or as a windows service.
The metadata definition contains:
Description of input files (Fixed length, CSV, Auxiliary Layout)
Sample definition of field attributes:
Record & field definitions, record & field logic, and record sequences
Definition of regular expressions
In metadata, users may create definitions of attributes like records, lengths, scale of records, local region settings, data variables definitions etc. Data is validated according to these definitions in the metadata. They can be changed easily as metadata is used as a configuration file. Data is validated in three steps – field (attribute), record, and whole file. All information, inconsistencies and possible errors are kept in log files for further processing. Together with validation described in the metadata file, users can add their own functionality if it is defined in the class.
Build-in Java language for scripting
The Dataflow Validator supports Java language inside metadata in order to create advanced validation routines or to post-process data or to utilize counters.
The Validator can simultaneously handle multiple data files, the maximum recommended limit is 20 files (concurrent threads).
Upcoming improvements and enhancements:
- Anonymizer – feature that anonymizes data for testing purposes.
- Test data creator – feature for generating test data files.