Project Background
The client has custom REST API services that processes the external feeds, execute data quality rules, transforms and loads content into their data mart. End to end lineage is not available in the current systems. Requirement to view end-to-end lineage from a field on an incoming data feed to a column in their data mart, enriched with business metadata. All metadata sources required for supporting lineage were hosted on their AWS VPC, while DGC was on the Collibra Cloud, and as per the enterprise security standards, no direct inbound connections could be made to the enterprise VPC (i.e., no Collibra Gateway deployment)
Client wanted a centralised notification for approved metadata changes, that could be consumed by downstream applications. Custom processes for governing the business and technical metadata including definition, modification linkage and approval had to be supported with entitlements managed both at a functional area and data domain level.
Solution
- Solution Integration templates were designed to load the technical metadata from the REST api interfaces and from the related data mart hosted in AWS Relational Data Service. The integrations had to use credentials stored in Hashi Vault for connecting to metadata sources. The integration endpoints needed to be authenticated using HantWeb.
- For supporting centralised notifications, a generic workflow was designed to detect new assets, changes to asset attributes (modification or deletion) and write a JSON message to anAmazon SQS queue, which is then consumed by downstream applications.
- Custom workflows were designed to propose and approve business assets such as Business Term, Glossary, Vocabulary as well as manage the linkage between the technical assets and the business assets. Additional workflows were defined to promote a business asset from local to the enterprise glossary and move business assets between functional area glossaries.
- DevOps build and deploy scripts were also developed.
- Cloud Formation Scripts were designed and developed to instantiate Mule servers and the Mule Management Console on AWS, and to help scale the Mule infrastructure on demand.
Outcome
- Provided the client the ability to improve the users’ trust in data by providing the needed technical lineage across their data marts right from the external feeds through the applied data quality rules and transformations.
- Provided the users a unified interface for downstream applications to consume approved metadata changes from Collibra.
