Considerations and pitfalls for a successful data migration
Adopting a cautious approach to the data migration part of a system upgrade or application replacement project is the smart approach.
There are slightly different things to be aware of depending on whether you are a business owner/project manager overseeing the project or a business analyst/IT developer doing the job. Therefore, we have made two separate lists of important considerations.
Here are our points that you as a manager should consider ensuring that your project will be a success:
As a project manager or business owner
1. What is your migration strategy?
Work out what it is you want to migrate and when. Remember, you are in control. Get a clear scope ready and agreed with your stakeholders. Think business logic and business value.
2. Don’t underestimate the importance of the data migration and what’s involved
Consider this when you are determining which new application to acquire and which data to migrate, when and how to proceed (time, cost, effort).
3. Closely understand the data requirements of the new application
Does it have a robust data migration documentation, interface and methodology? Be sure to consider the quality and availability of the data required by the new application.
4. Break the migration down into manageable chunks
Think about how you can stage and phase the different deliverables to achieve early success or see signs of trouble early. Think agile.
5. Flexibility is important
Working in a way that can accommodate changing requirements as the migration progresses can make all the difference. Again, an agile approach will help you.
6. Have full visibility and auditability
Make sure your team applies a structured and documented approach that will enable you to maintain oversight, measure progress and understand any issue that may arise.
7. Automation is important because it cuts down cost
Make sure you automate as many manual processes as possible. Humans will make mistakes. Use automated, repeatable processes where possible.
8. Commit resource and accept this cost up front
There must be internal commitment and involvement, you can’t outsource it all. The data migration will require your best people and be more difficult than you expect.
9. The new system is different from the old
Remember you are trying to move to a new system because it will give you new functionality. The new system will only deliver value if it has clean data and if these are applied correctly.
10. Monitoring and transparency are important
Data migration is a big decision as well as a big risk. You need clear visibility of where you are in the process, so you can understand any issues and make informed decisions.
As a business analyst or IT developer
If you are more hands-on as part of the team delivering the data migration, here’s a set of important points for you to consider:
1. Identify and engage with business users
You need to identify the right business users from the outset. People who knows and understands the business data. Bring the people who’ll be using the data into the migration project. After all, they’ll be the ones operating the system once it goes live.
2. Specialist needed
One of the most important and common problems with data migration is the fact it’s often disregarded or underestimated by management; whose representatives don’t pay enough attention to the importance of data migration projects. They consider it one of many routine tasks which IT employees should be able to pass through without any special help nor support.
Unfortunately, this is not the case. Data migration isn’t just a matter of IT, it’s a process involving almost every employee across the organisation. It requires engagement, not only from IT, but also from experts from other departments, data users, etc. In addition, it is also commonly necessary to allocate powerful system resources to data migration, in order for it to complete quickly and efficiently.
3. Unrealistic scoping and budgeting
The immensity of data migration tasks might be overwhelming at the beginning, but it’s necessary to identify and describe everything that must be done. Omitting this is a very bad idea, because it doesn’t give a proper insight into the scale of action and, also, its cost.
It’s extremely important not to underestimate the cost of a data migration – looking for savings at this stage is suicidal. The same is to underestimate scope – the number or intensity of things that must be done – makes data migration longer than predicted and, as a consequence, much more annoying to data users across the organisation.
4. Wrong Tools
The choice of tools should be a part of any data migration strategy, but the fact is that managers from different organisations don’t pay real attention to tools designated to support data migration.
In many cases, the tools used for data migration are the ones the company already owns. They’re often not designed to real data migration; but rather made to support totally different uses, for instance, Data Warehousing. Purchasing new tools for data migration isn’t what many organisations consider right – but is what should be done.
5. Data governance and organisation
Who decides what is in the project and business organisation? Decisions must be made as to what data should be moved from system A to system B and what needs to be done if data quality is poor or required data doesn’t exist. To know who signs off on what is crucial to progress.
6. Poor data quality in a legacy system
Once you understand the requirements of the new system, it’s important to perform an “as-is assessment” on your existing data before embarking on a data migration project.
Understanding the quality of existing data in a legacy system is a huge pitfall that companies often don’t spend enough time on.
It is worth considering whether the existing data will support the new system? What is missing? What can be done to fix what missing, and will there still be data you will not be able to create?
A detailed assessment makes it easier to estimate the amount of work required to migrate legacy data successfully.
7. Neglecting to validate and redefine business rules
Your company’s business and validation rules may not be current or applicable in the new system. As systems change so does rules and processes.
Often too little time is spent agreeing on business rules, much less making sure the data complies with the business rules. You may think you have a business rule, but does your existing data match, map, or comply with that rule?
In addition, auditors must be sure that data moved from a legacy system to a new system has been validated, especially when a migration involves critical information such as financial, inventory, and payroll data.
8. Failure to validate and test the data-migration process
Don’t save this step for the end. You really must make sure that you’re validating and testing throughout the process.
Think about how are you going to test the data? Who will test and evaluate it? Who will sign off on the quality of the migrated data? And who’s the ultimate user of the data?
This process must be built into the lifecycle of the project, but unfortunately, companies often don’t spend enough time aligning the data testing, validation, and migration cycles to the project timeline.
As with any list no matter how long it is never complete. There will always be project specific issues to consider coming from the systems and organisations involved.
Use this list and others before you get involved in your first or next data migration – and good luck.