Our approach to data migration
An agile methodology based on business objects
as the unit of migration.
Agile data migration approach
Project management methodologies are the blueprint of a project, thoughts to guide the project team to the successful completion of its task.
Data migration is a complex process, requiring a robust methodology and good supporting tools to deliver a high-quality result as fast and cost-efficient as possible.
The agile method implies an iterative and incremental method of management. It focuses on helping teams in an evolving landscape to maintain focus on rapid delivery of business value. Agile is based on continuous improvement, flexibility, input of the team, and delivery of results with high quality.
A managed environment
By using migFx you insert a self-sustained environment between the source and target system. In this, you can iterate, enhance, transform and validate your data step by step in an agile fashion.
- Contains all the functionality you need to map, transform and validate your data
- Makes it fast to start, and easy to progress in controlled increments
Our data migration approach consists of using agile sprints with multiple iterations that provide incremental improvements.
You can use our data migration approach to scope, plan, execute and document your migration choices and tasks.
Why agile development for data migration?
It is well documented that complex data migrations often fail due to a lack of awareness, resources, time, tools, and data quality.
We like to add to this list: Lack of methodology or – more correctly – using the wrong methodology. Its most often the use of traditional waterfall and IT software development approach which doesn’t work for data migrations.
Often the source systems are old, large, fragmented, undocumented, and complex and the new system requirements not yet fully decided or understood.
As data migrations are infrequent, the skills needed are in short supply. Therefore, many issues and requirements are discovered late in the project, sometimes too late to correct them without delaying the implementation.
The agile approach requires that the iterations (sprints) of development align with an overall vision and a feature backlog, it also requires that the development of code fits within an overall architecture in order that the delivery is automated as far as possible.
migFx provides that architecture and a large part of the automation to enable the data migration project teams to operate in an agile manner.
The features should be aimed at delivering usable instances of migrated data that support the development and testing schedule of the main programme, in this way an agile approach can be used for the data migration project even if the main programme is working on a waterfall approach.
The user stories should be aimed at developing the detail of the extract and transformation rules, the architecture of the solution should be in place and provide a stable environment for that agile development of the rules.
migFx provides such an architecture, one that will allow you to progress in an agile way without ending up having to redo a large part of the solution at a later stage.
Working within a defined architecture best supports the “start early before the final design of the target is known” approach.
For the start early approach to work you need to work within a designed architecture, so you know what the outline of the final result will be if not the detail.
migFx supports agile
The typical data migration project starts with no firm foundation.
The structure will gradually have to be established and built during the project.
Starting without a structure cause difficulties and delays further down the line and – in some cases – the complete rework of the entire migration solution design.
If the overall scope of a migration is viewed as achieving full business object “coverage” and full ”depth” in attributes and quality we can then illustrate this by having coverage along the x-axis and their depth along the y-axis.
migFx provides a structured environment that enables the effective application of the agile approach to data migration projects, which otherwise would have to be designed and built by the development team before any productive work can begin.
No data migration exists in a world of static and well-defined requirements. The reality is a moving target, as the target system itself is undergoing changes – because the data migration uncovers the gaps between the source- and target systems.
Using traditional tools (such as ETL) many projects bog down over time as both mapping and implementation struggle to keep up with changing requirements. Often the implementation drifts away from the specification – thereby losing the only instrument available to ensure the data migration is aligned with the business requirements.
Using migFx, the process of modifying the mapping, generating the code, and iterate the migration is fast and agile. migFx ensures consistency between mapping and implementation.
The mapping is the implementation – always relevant, valuable, and the only place where requirements are implemented and documented.
The rich support for cross-reference, validation, and what-if scenarios in the mapping enables constant sight of the ever-moving target.
Uniform code and handling of events during the migration iterations ensure that the implementation always behaves in the same way throughout the lifetime of the migration project – regardless of the changes made.
No mess agile
Agile development, if not implemented correctly, can lead to an unstructured tangle of code. The structure imposed by migFx prevents this while still allowing fast iteration of development and execution of a partially developed solution.
The sprints in an agile development can then be defined as a small number of attributes across a number of business objects or a larger number of attributes across a single business object to gradually fill out the whole scope.
Agile vs Waterfall
The main difference between waterfall and agile methods is in the goals; the waterfall method wants to get everything right the first time, and agile methods want to get things released quickly.
Learning through trial and error are features of the agile mind, while build to specification is the waterfall way.
Differences are visible in the way the different approaches lend themselves to adaptability, documentation, testing, and collaboration in support of the different goals.
Let’s put it this way: Waterfall moves through different stages or phases like design, develop and test, implement only once while the agile approach has multiple such iterations over the project life cycle.
Incremental progress
Complex data migrations are often – well, complex. When using the traditional waterfall approach, it typically takes a long time from the initial specification were made to any results are seen.
Often many design and implementation choices are written too early from which it is difficult and risk-prone to deviate from once things have changed.
Using migFx, it is straight forward – and indeed best practice – to get operational extremely fast. You don’t need a complete overview or full understanding to enter an agile loop.
Just start immediately by mapping just a small part of one or more Business Objects, generate the code and start iterating – early results are there.
The consistency and uniformity imposed by migFx enables the incremental addition and enhancement of Business Objects.
This process is perfectly aligned with the most natural and indeed unavoidable way ahead in any data migration project.
You will find your data migration team continuously and confidently iterating the agile loop – typically many times a day.
This incremental approach to the challenges takes the quality of the data migration to an entirely different level – and leaves you with a data migration team feeling empowered and successful.
Pratical Data Migration (PDM)
Looking for a methodology for your data migration project?
PDM has been developed by Johny Morris. It is the only publicly available end-to-end data migration methodology. It takes care of a project from contracting in of suppliers, through source data analysis, data mapping, data quality, extract transform and load design, migration execution and legacy decommissioning.
“Whatever you need from a data migration PDMv2 has the answers and migFx fits very well with it.”, Johny Morris.
