Data Migration for CRM and ERP systems
No enterprise remains static; they are continuously evolving and changing in response to internal and external drivers. Similarly, the IT systems that support the enterprise need to evolve to keep up with the new requirements, this may involve replacement or migration to new systems. All this is in addition to the regular maintenance updates and enhancement requests. Central to supporting any changes, whether they are large or small, is the need to maintain full and precise data integrity. This necessitates the need to understand the data and underlying metadata involved. However, this is easier said than done especially where the metadata is not held in business-friendly terminology, nowhere is this truer than in ERP and CRM systems.
Barriers to effective data management and migration for ERP and CRM systems:
- What are the data/tables?
- What data/tables are important?
- What is the data lineage?
- What transformations are applied to the data as it moves?
- Who has ownership of the data?
- Why is the data of importance?
The solution to the problems of a successful data migration exercise is to understand the data and their mappings:
- Determine the relevant metadata. The normal method of finding metadata associated with systems is to reverse engineer the data model from the RDBMS’ System Catalog. However, whilst this is possible, the results are of almost no value.
There are a few reasons for this:
a) The System Catalog contains no business names for tables, attributes and other objects rendering it difficult to know what each object means.
b) There are no primary and foreign key constraints defined in the database so trying to find out how tables are related is virtually impossible.
c) The size of the data model, for example an SAP system has over 90,000 tables, means that a data modelling tool would be swamped by the amount of information extracted. And with no relationships or business names available the result is an unwieldy, unnavigable mess.
- Focus on the portion of the ERP/CRM application affected. For example, SAP ERP applications contain close to 74,000 tables and over 800,000 fields. However, typical projects involve mapping to only 100’s of tables fields.
To overcome these barriers use specialist metadata discovery software which can find and extract the useful metadata – business names, relationships and more, and which can then be used to analyse and subset it into models that represent what is needed in the context of a project.
- Find out who is interested in the data and why. It is the job of the data steward to answer these frequently overlooked and assumed, questions. Any lack of understanding contributes to the proliferation and duplication of data.
- Keep data mappings up to date. The data mappings present their own sets of problems too where the mapping document is most likely to be out of date with unknown, incorrect, duplicated, and or redundant data mappings.
a) Start mapping early in the project and keep going to the end. Teams must start mapping needs early in the project, but they also need to realize this process will likely not be complete until the very end of the project.
b) Use automated tools to both reverse existing data mappings and document future mappings. This will provide for better accuracy, re-use, and faster development.
c) Define levels of mapping at the appropriate stage:
i) System to System – scope of project
ii) Object to Object – affected and involved metadata
iii) Field to Field – mappings and their transformations
iv) Value to Value – explicit value-based mappings
5. Read mappings specifications carefully. Read through the mapping specifications judiciously bearing in mind that mappings are likely not be complete until the very end of the project. Treat the mappings specifications like a contract: approve only if you agree to be bound by it.
Not following this simple guidance means that enterprises spend extra time and resource trying to work out what objects and mappings they need for a particular project using the largely manual methods they have employed in the past. These can hold up realisation of project benefits by delaying delivery and reducing the trust the business has in the data. Each project that needs information from one of these systems will go through the same process again and again unless a different approach is taken. Work smarter: use tools to help you automate, accelerate, and improve the validity, confidence, and trust in your data.
Interested to hear more about Making sense of your ERP and CRM systems metadata using smart modeling? Watch our webinar recording here.
Blog content provided by Sandhill's Jog Raj and Silwood Technology documentation