What is a Data Catalog?

A data catalog is an inventory where you can store all your Data Assets in an organized way. Data Catalog is not really a new thing, in the past, people have managed data assets in various ways. But, in the last 5 years the usage of Data Catalog, using packaged software has really gained a lot of traction.

So Why we need a Data Catalog?

In 2010, there were 2 zettabytes of data created, copied, consumed and in 2020 the figure increased to 59 zettabytes and it is predicted it will increase to 150 zettabytes by 2024. Not only the data but also the types of data, tools to store, and process the data has increased. Even smaller companies are becoming more data-driven, meaning they started taking strategic decisions based on Data Trends. Now when all these are happening, it is very important to govern the data properly and Data Catalog can be the best way to implement it.

What can be done in a Data Catalog?

• Data Discovery: You can search for all sorts of Data Assets (Technical, Logical, Business) in one place.
• Data Lineage: One of the most important things that the organization need these days is to understand the flow of their data assets.
• Asset Storage: Various assets, mentioned in the previous sections (Technical, Logical, Business) can be stored in a systematic way.
• Semantic Discovery: To be able to understand semantics associated with any assets by giving different metrics to discover the data and relation.
• Business Ontology: To represent the Business Entities and processes and how they are related to each other using a common language.
• Data Quality: To see how reliable the Data Assets are to give confidence to the user by associating the Quality Score to Assets.
• Policies: A very crucial aspect of managing data with a combination of the user communities.
• Compliance: Metadata is a great way to stay compliant with the latest government rules & regulations. To name a few: GDPR, HIPPA, BCBS 239…
• Versioning Of Metadata: While metadata is important, versioning it will give also plenty of advantages to see the trend also report accurately.
• Reporting Of Metadata: Since all the Data Assets are stored in place, it is easier to create reports which have meaning.

There are also few other advantages that people realizing these days like, it can also act as a data register for Data Virtualization. Also to store artifacts like Architecture Diagrams.

I have used various Data Catalogs so far and I realized it got immense power to solve, some of the biggest problems organization has. A great way to start can be to explore some Open Source alternatives like Apache Atlas, Project Amundsen from Lyft.

But if you are already using packaged ETL products like Ab Initio, Informatica, IBM they also have Data Catalog Offerings. Specific metadata products such as Collibra also used widely these days. If you are in the public cloud space there is interesting work happening these days. One such example is Azure Purview which is in public preview now.

Note: Above mentioned products might not have all the features out of the box, you really need solutions to develop Metadata Architecture.