“When will the data migration deliverables be “done?”

Data Migration doesn’t fit well into project tracking schemes. Unsurprisingly, the frustration of stakeholders eventually becomes palpable on every project.

If you ask me — the functional master data guy — when data migration will be “done” then the honest answer is clear: “The day before go-live.”  When will the data migration functional specification be done?  Same answer.  Go ahead and ask the technical team when the data load program will be finished.  “That depends on the functional specification.  Oh, and you’re not going to change anything over time, are you?”  For some reason, replies like this reliably produce eye rolls.

If you’re part of a non-productive conversation like this then you’re having the wrong conversation.

Here’s the simple reason for what appears to be over-the-top hedging:  data migration is iterative, and it must accommodate change over time. Significant change in the beginning, and carefully-considered change later on.

The result is that data migration progress doesn’t fit neatly into a project plan or make for a pretty Gannt chart.  And hammering data migration deliverables into typical “check the box” reporting doesn’t provide transparency to project stakeholders.  The original question — when will this be done? — is kind of important and deserves a proper answer.

Another challenge is that conventional project methodologies demand that functional specification documents be “done” at the start of the Realization phase.  After all, somebody has to create something to load the data.  A document is necessary.  Yes, there’s wiggle room and prioritization, but I’ve never seen this happen without a discussion of the definition of “done.” Let’s just say it up front: it’s never “done” because it’s iterative and we necessarily learn over time. Let’s embrace the reality and at the same time embrace our responsibility for transparency.

A proper answer — data migration transparency — can be found by monitoring key data migration milestones, informed by experience.   If you’ve survived being responsible for data migration then you have a sense for the timing of events that either produce a warm and fuzzy feeling or raise a red flag.  Those experienced-based events are the foundation for producing a different form of “check the box” reporting to communicate the progress of this most critical workstream.

Reformulate the universal question as:  “Are we on track for success with data migration deliverables?”  That’s a better question, and one with an informative answer.

Key Steps

Before enumerating a list of warm and fuzzy indicators, as context here are the key steps for each business object:

• Analysis
• Extract
• Transform
• Validate
• Load
• Reconcile

For your data migration project, there should be a data migration architecture / design document that explains the general approach to be taken for each step.  For example, what does the “Reconcile” step look like, exactly?  Will the data owner accept simple row counts (I gave you 20 to load and 20 were loaded) or will row-by-row and field-by-field reporting be required?

The point here is two-fold.  First, to continue with the “Reconcile” step as example, two reporting extremes were mentioned, with equally extreme levels of effort. Up-front understanding and agreement is important.  Second, while a general definition of each step exists for the project, it’s important to realize that there will always be differences in approach by business object (e.g. product, location, supplier, customer).

For each business object there will be a data migration functional specification document, owned by the functional person responsible for the business object.  That’s the central document consumed by all participants in data migration activities (legacy data, extract/transform, technical).

For that reason, progress is tracked by business object and, when appropriate for transparency, what is tracked may be slightly adjusted by business object. For example, if a business object requires data construction or data quality mitigation, you might want to include a tick mark or progress indication  for that activity for that business object.

Data Migration Progress Indicators

By business object (i.e. for each conversion functional specification document), the milestones that I like to track for master data conversion, providing visibility to integration-test-cycle readiness, are the conventional ones I’ve seen on projects.

There’s a focus on the load step because the only reliable indicators of progress are demonstrated and repeatable results, with initial “success” determined and declared by the owner of the functional specification document.  If there’s one reliable mantra for data migration success it’s this: Load early and load often.  It’s all about repetition over time, and accommodating reasonable change over time.

1 – Object Definition / Prototype

• In sandbox or development data migration client, load a few rows of manually created sample data.
• This can be accomplished far in advance of known requirements.
• Success demonstrates prototype load functionality for a business object.
• Simple result informally validated by person owning the conversion functional specification document for the business object.

This exercise and indicator may seem simplistic, but the value can’t be overstated whenever there’s any unfamiliarity with the business object or with the tool required for loading a business object.

Load step options include these tools, and you will need more than one as all business objects are considered:

• SAP Data Services
• SAP S/4HANA Migration Cockpit
• Legacy System Migration Workbench (LSMW)
• Extended Computer Aided Test Tool (eCATT)
• Custom Development Object / Load Program

2 – Unit Test

• In a development migration client, load a few rows of manually created sample data.
• This can be accomplished, in part, in advance of a “final” functional specification document.
• Success demonstrates load functionality for a business object with baseline configuration.
• Simple result informally validated by person owning the conversion functional specification document for the business object.

Achieving this milestone absolutely requires real-time and high-touch collaboration between the functional person who owns the functional specification document and the technical person creating the data migration load step. These two are partners, and their collaboration is the key to success. If you don’t see visible signs of it then you should be very worried.

The functional person provides necessary inputs to success (verbal guidance, near-real-time adjustments to configuration and sample data) and determines when initial success is achieved. In other words, the functional person checks the box for this indicator, or is intimately aware of what is preventing checking the box.

The result is many fast iterations (as many as needed). This is how we achieve speed and quality.

3 – Functional Test

• In a development unit test client, load a few rows of manually created sample data.
• In a development unit test client, load a representative subset of data supplied by data transformation process. Data must represent different flavors that may exist (for example, for Business Partner, coverage for all Business Partner Groupings).
• Result validated by person owning the conversion functional specification document for the business object.
• Success demonstrates load functionality for a business object with baseline configuration and real-world data.
• Result reviewed by data steward responsible for business object.
• Result validation includes use of master data in a business process.
• Result determined to conform with functional specification document; exceptions noted.
• Includes preload-validate step, to confirm that data aligns with configuration and master data in the target system.
• May or may not include reconciliation reporting.

4 – Full Load (or … almost)

• In a development unit test client, load a full or significant subset of data supplied by data transformation process.
• Result validated by person owning the conversion functional specification document for the business object.
• Result reviewed by data steward responsible for business object.
• Result validation includes use of master data in a business process.
• Result determined to conform with functional specification document; exceptions noted.
• Includes preload-validate step, to confirm that data aligns with configuration and master data in the target system.
• Should include reconciliation reporting.

Load Early, Load Often

Notice that all of the above occur before Integration Testing in a Quality system. In my experience, these are reliable indicators of readiness to begin Mock Conversion and Integration Testing.

Throughout project Realization phase, as requirements change (a new field here, a new field there … defect corrections), the indicators should be reconfirmed, with an appropriate level of formality.  Again, the only reliable indicators of progress are demonstrated and repeatable results.

Show me the data!