USA Telecom
Using metadata-driven automation for ETL conversions as part of leap to Big Data
CLIENT PROFILE
The customer is a multibillion-dollar global telecom
company headquartered in New Jersey that provides residential and business communication solutions.
CHALLENGE
The customer wanted to move toward a Big Data architecture and be able to integrate metadata from such internal systems as Kafka, Flume, Hadoop, RDBMS, Teradata, flat files and MongoDB. They also wanted to migrate to Talend as the ETL platform, which would require the conversion and validation of a large number of complex graphs within a short timeframe and with limited documentation around the legacy graphs. Unit, system and Soak testing also was required before a complete transition, which is a chaotic and tedious process overall.
SOLUTION
erwin executed a proof-of-concept (POC) to demonstrate the feasibility of automating the conversion of legacy graphs to Talend using erwin’s data connectors and partnered with Talend to architect and validate the solution design. With Kafka as the messaging architecture, it was necessary to read the Kafka component to move the data to the Talend platform, so erwin built intermediate convertors to facilitate a two-way reading mechanism between Kafka to Talend and vice versa. Through this code logic, it was possible to read the byte array of Kafka in both a specific and generic manner.
– Realised efficient and on-time conversion of 30+ legacy graphs compromised of approximately 1,000 components to Talend, producing significant cost savings on licenses (estimated in the millions)
– Integrated data from internal systems, including Kafka, Flume, Hadoop, RDBMS, Teradata, flat files and MongoDB
– Streamlined the ETL conversion process, making it predictable, consistent and easy to adopt best practices and design
– Provided robust technology, a futuristic roadmap, and the ability to scale according to the changing technical landscape and market advances