Data Migration: Is the transferring of data from one system to another. Think of this as “data on the move”.
Data Conversion: Is the transformation (or translation) of data from one format to another. Think of this as “data changing state”.
Data migration is used when upgrading the data model, acquiring a new pipeline of data (additional data feed), or moving from one system or database to another.
Data Conversion is used to extract data from one source, transform it, and prepare for load into another system.
We are working on multiple projects that include massive data conversion and data migration activities. Millions of records will be converted and migrated, and accuracy is critical. When the process runs, it will look simple. But the prep work is key. Every data element must be reviewed to determine the unique identifiers (what are the keys?), the issues with these identifiers (oh no, were the account numbers reused and no longer unique), the expected landing position, and the meta-data (what is the definition for this data element?).
For people who aren’t involved in the process, I’m sure they wonder what the big deal is. Take some data – convert it and migrate it. Aren’t you done yet? For people who are involved in the process and do it well, they know this can take weeks (maybe months) and every step needs to be perfect. Failure to understand the current state of the data, failure to validate the business rules for the migration, or failure to test thoroughly can result in corrupt data. And who wants to receive a bank statement with someone else’s data (name, address, account activity, or balances)? Okay, maybe you’d like a peak into someone else’s data. 🙂 But I’m betting you don’t want someone else to receive yours!
Now to make it real for anyone not working in the data teams today…
Think of standard chores you do at home. Taking all the little notes you wrote yourself and consolidating them into one list. Data conversion – you’re an expert. Now take that list and move it from paper into your computer. Data migration. See how simple it can be? And rewarding? And yet, it all takes time.
To all the unsung heroes working on data conversion and data migration projects, I salute you. You inspire me. It’s not glamorous work. It’s not too much fun. Okay, maybe it is for geeks. But it sure is important.