Why do you need Unified Meta-Model for Metadata Management
Introduction
Every organization has its own way of managing data. Some take it seriously and have long-term plans, some play around with the latest tools and technologies and some are time-bound by regulators. You got the idea !!!
To become a truly data-driven organization it takes a lot than just to have a data strategy or rolling out a newer software stack. The secret lies within holistic governance and management of the data and a great way of addressing this is Metadata Management. It is identified as a knowledge area in DAMA, and this topic actually glues other areas of data to have a proper Data Management Practice.
Why Metadata Management?
We are living in a world where we see concepts, techniques, and tools pop out every single day, but if you try to understand what they do, you will realize the fundamentals are still the same, “how they do” is a different topic and is also more towards the architecture side of things. Because of the abundance of concepts, applications & tools as well as data, it is very easy to get overwhelmed with it and lose the oversight of the Data Estate. Managing metadata and mapping them to processes, assets, policies & people can give you a holistic overview and semantics. This not only helps the management to take an informed decision but also helps business and IT to have a hold of things and stay in control. Metadata Management is really important for Risk Management Activities and can help to minimize Risk Exposure as well as help in meeting the compliance requirements.
Where to Start & How to Start?
Metadata Management is not something new. Management of metadata was always there behind the Spreadsheets & SharePoint. So, it is possible you are doing it already but not knowing how it can bring value to all the data initiatives going in the company. With the emergence of modern-day catalogs, there is already a lot of hype going around it. You will be tempted to jump into a product that can solve this, but in reality, this is not always true. There has to be some business case, from which you could start a program and slowly build on top of it. Defining Critical Data Elements can be a great way to start. To drive a program, also consider having a meta-model. There are numerous challenges when it comes to managing business metadata and technical metadata and a unified meta-model can help a lot in this case. Meta-model can bring in a lot of value for all the Data Management initiatives. It not only helps to manage your metadata in a structured way but also acts as a source for sharing metadata to applications & processes.
The Unified Meta Model:
All the Data Catalogs have a meta-model, some are relational and some are non-relational. But at the conceptual level, they all represent the same core fundamentals. Things like Datasets, Data Element, Data Domains, Systems, Application, Classification, Business Glossary, Ontology, Technical Lineage, Business Lineage, DQ Scores, and the list continues. So it is also a great idea to create your own meta-model to store metadata the way you want to have it. This can be really useful to process metadata to a central Data Catalog or to Custom Interface within the organization. The custom model can work in parallel with the Data Catalog. For Business Metadata, it is also really useful to store all the metadata in a systematic way rather than maintain the excel spreadsheets.
Just to illustrate things, I have prepared a Conceptual Model to hold Technical Metadata.
The above metamodel also gives a framework to the developer community to build application which supports metadata, lineage, etc. You can extend it with business metadata to enrich further.
The above model has components required for Data Discovery, Data Governance, Data Classification, Data Privacy, Data Lineage, etc.
What are the advantages of the Meta Model:
End to End View: You can create a knowledge graph of all your asset and get enough insight into how a business process or entity is linked to IT. For example: When an application is running as per schedule, there must be a job that is running behind the schedule, the job is responsible for creating datasets that are required to create some business value.
Easy Integration: There is also an advantage if your organization is having many application, technologies, custom processes, it is something very difficult to address them with a catalog and needs good meta-model. You can develop hooks from applications to push data to this model.
No vendor lock-in: The meta-model gives a skeleton to manage metadata and you can actually choose or migrate to your choice of catalog. The meta-model can act as a single version of the truth.
Versioning: With versioning of metadata, you can do time travel to understand what has happened in the past and use the data for regulatory reporting or management reporting.
What are the downsides?
Of course, this is not all shiny, It would take a lot of effort to scope the meta-model and decide what attributes you need in them. You need to decide what level of granularity of metadata you need to store ( Business, Technical, Operational). You need to make sure your application is able to push the required metadata to this model. And last but not the least , budget & time, you need to invest a lot in these initiatives and give this time to mature.
Conclusion :
I have written this article, to help to understand the benefits of a Unified Meta-Model for your metadata management initiative so that you can make informed decisions while starting with the metadata journey. Remember tools are there to facilitate things faster but fundamentals are really important. Understand all the details, do the requirement gathering properly, target what you want to achieve and how you want to achieve, who are your users, and then decide which direction you want to proceed. There are articles related to The ISO/IEC 11179 norm for metadata registries which can give a lot of insight into Data Element which is a major part of Metadata Management.
