What is Audit and Reconciliation for Data Migration

What is Data Migration Reconciliation?

Data reconciliation is a process to verify that data from a legacy source system was migrated according to specified requirements. In essence, it confirms "the data are still the same."

The primary challenge involves transformed or changed data, making simple one-to-one comparisons insufficient. Examples include merging duplicate customers, redefining products, or bundling services differently.

Audit vs. Reconciliation: What's the Difference?

Reconciliation:

• Compares two sets of records to ensure agreement
• Verifies that data leaving an account matches what enters the new system
• Part of internal organizational processes for accuracy verification

Audit:

• A formal check of data, processes, financial accounts, or policy adherence
• Examines reporting, accounts, documents, and activities against regulatory requirements
• Conducted by internal or external auditors

Why is Data Reconciliation Important?

During migration, errors can occur through extraction, mapping, transformation, or load procedures. Technical failures may corrupt data, causing:

• Missing records
• Incorrect values
• Duplicated records
• Wrongly formatted values
• Broken relationships across data elements

Data reconciliation helps by extracting accurate information, producing consistent datasets, and integrating automated processes with meaningful reporting of divergences.

Key Techniques for Data Reconciliation

Associated Terminology:

• Indicator or Control Point: The data element chosen for reconciliation
• Gross Error: Reflects bias errors, measurement failures, outliers, noise, and spikes
• Observability: Which indicators can be determined based on constraints
• Variance: Measures variability of an indicator
• Redundancy: Determines which measurements could be estimated from other indicators

How to do reconciliation during a data migration

A step-by-step approach:

1. Pre-Migration Data Assessment: Review source data for completeness and identify quality issues
2. Define Reconciliation Metrics: Establish key metrics such as record counts and sum totals
3. Data Mapping and Transformation: Map source fields to target system and transform as necessary
4. Perform Trial Migration: Conduct trial migrations with data subsets
5. Execute Data Migration: Migrate full dataset while monitoring for issues
6. Post-Migration Reconciliation: Compare source and target data using defined metrics
7. Resolve Discrepancies: Correct identified issues and reconcile again
8. Final Validation: Confirm accuracy and completeness before sign-off
9. Documentation: Record the process, findings, and resolutions
10. Continuous Monitoring: Set up mechanisms ensuring ongoing data consistency

The challenge involves using systems independent of the process being verified.

Best Tools for Data Reconciliation in Migrations

Hopp's Approach:

Reconciliation broadly covers ensuring that all in-scope legacy data is either migrated or discarded per requirements. Simple reconciliation may involve counting: comparing source invoices against discarded invoices and delivered invoices.

More advanced reconciliation becomes complicated with extensive transformations. Hopp exposes interfaces to inspect migrated data and collect audit data representing actual migration results. By implementing these interfaces, projects collect audit data for each business object during migration, with Hopp Runtime storing this data alongside business objects.

Hopp also exposes an Audit Result interface called when operators initiate audit jobs from the Portal Operations interface, providing all collected audit data.

Solution: Hopp delivers a mechanism to collect audit data representing actual results. Migration projects establish reconciliation of audit data with expected results outside Hopp.

Questions and answers