Data Cleansing & Migration

What is Data Cleansing & Migration?

Data migrationData Cleansing is the activity of removing duplicates, adding missing information, ensuring consistency, and making corrections.

For example, let’s say you have a Client database and people from all over your organization enter data into it. For Client Company ABC, there are three entries:

  • Lily Smith, Payables clerk
  • Lil Smith, Office Manager
  • Lillian Smith, CEO

Is this the same person with 3 different records or 2 or 3 different people? In this example, the Data Quality technician would either need to investigate further and make the necessary changes or make a judgement call based on defined data cleansing rules and procedures and take action according to those rules.

Sometimes data needs to be cleansed as on ongoing operations activity or as a one-time project. Examples of projects include situations when new data is introduced in bulk, such as an acquisition of a new company that has some of the same clients, for example. Another situation for project-based data cleansing is when there is migration to a new system.

Data migration can mean migrating from one technology to another, such as hard copy in 3-ring binders to digital scanned files. It could mean a migration from one software system to another, one data base structure or product to another and on it goes.

What is its Purpose?

Data Cleansing restores and/or maintains data integrity which is key to having confidence and trust in the data. Clean data ensures that accurate results are used for decision making and time and credibility are not compromised.

Data migration ensures existing data is converted into a new system so that no data is lost or misinterpreted in the transition.

How is it Executed?

Data Cleansing can be executed directly into the live data or on subsets of data extracted from the live data and, once cleansed, merged back into the live data. Various software tools are used, a common one being a spreadsheet.

Data Migration is driven by the conversion taking place. Perhaps, scanners for paper-based files are used or complex software applications especially developed to convert from one software application to another.

Consequences of Inadequate data cleansing and migration

Inadequate data cleansing can result in an organization relying on unreliable data for its day-to-day operations which results in wasted time, loss of credibility, and poor morale. It can also result in decisions based on faulty information.

Inadequate data migration can result in delaying the start-up of a new system, lost data, or misinterpretations of data from the old system to the new one. The results can impact the implementation of a new system or platform, or access to new data.