Metadata Platforms Evolve as Foundational Context Layers for AI Agents

Key Quotes

"First full context of metadata is important for people to understand their data and, you know, do things with it. The second thing we realized is data as it got democratized self service, right, and nearly one third of people within an organization are in some way or shape or form they're data practitioners, they're using data for their day-to-day decisions, and we saw that these people were disconnected from each other and they were not collaborating with each other, which was the cause of many of the problems that we saw in the space of data. And then finally, the third thing that we saw is people, you know, who are exceptional with data were spending a lot of time on mundane, you know, tasks, right, of cleaning up the data, documenting the data, things like that. We saw that this needs to be automated, right, in order for people to make the best use of data and create outcomes with data, right. So that's that's our learning with which we started the OpenMetadata project."

Srinivas explains that the OpenMetadata project originated from observing three key challenges in data management: the need for comprehensive metadata context, the lack of collaboration among data practitioners, and the significant time spent on mundane tasks that could be automated. These insights formed the foundation for their approach to building a metadata platform.

"So, you know, data is transformative, right? It can transform the societies, you know, create new innovations. Harsha and I have been, you know, in data space for that reason. And, yeah, so, you know, you've seen, you know, what Hadoop brought to the data space, right? Before Hadoop, storing large amounts of data and processing large amounts of data was not possible, right? And through, you know, the solution that is based on commodity hardware, it made data accessible, right, for storing and processing large amounts of data, understand the world around us, right, through data. And so the potential of data is what, you know, I'm super excited about."

Srinivas articulates his long-standing passion for data, highlighting its transformative potential for societies and innovation. He uses the example of Hadoop to illustrate how advancements in data storage and processing have made vast amounts of data accessible, thereby enabling a deeper understanding of the world.

"The key phrase of the past ~2 years is 'context engineering.' What role does the metadata catalog play in that undertaking? What are the capabilities that the catalog needs to be able to effectively populate and curate that context? How much awareness does the LLM or agent need to have to be able to use the catalog effectively?"

This quote frames the current landscape of AI development, emphasizing the critical role of "context engineering." It poses fundamental questions about how metadata catalogs are essential for providing this context, what specific capabilities are required, and the necessary level of awareness for AI models and agents to effectively leverage this information.

"So, if you see LLMs are powered by data, right? Now, how do you power LLMs with data? Right? Context becomes very important. The meaning, right, that LLMs gather out of data becomes very important. So, OpenMetadata as a context for, you know, that provides context of data to LLMs is how we are empowering LLMs within the enterprise organizations to use AI, right, along with the data within the enterprises, right? It's a powerful enabler, enabler of LLM AI use cases, right, around the data within, you know, within the organizations."

Srinivas explains how OpenMetadata functions as a crucial enabler for Large Language Models (LLMs) within enterprises. He asserts that LLMs are fundamentally powered by data, and the meaning derived from that data, facilitated by context, is paramount. OpenMetadata provides this essential context, empowering organizations to leverage AI effectively with their internal data.

"So, for me, you know, there are two hats I wear. One is, you know, a technologist. The other one is a startup founder. The startup founder is, you know, really excited about the possibilities with technology. As a technologist, I'm always worried about the hype versus reality. In an OpenMetadata meetup, we were actually doing a demonstration of our MCP server that we had built, and then someone was demoing with Claude, right, the capabilities of OpenMetadata and how, you know, we expose tools that can be used by Claude. I was just amazed, right? Today, we have built all these... with Claude, you could actually combine the world of data or the internet, right, along with LLM capabilities with OpenMetadata where you can actually say, 'Give me all the banking terms,' and then it will give you banking terms, and then you can say, 'Hey, I want this, I don't want this term,' and then you curate your, you know, with natural language, curate your list of terms, and then you can say, 'Add it to OpenMetadata.'"

Srinivas shares a personal anecdote illustrating the practical impact of AI on user interfaces and workflows. He describes being amazed by a demonstration where Claude, an LLM, combined internet data with OpenMetadata's capabilities, allowing users to curate terms using natural language. This experience highlighted how AI can simplify complex tasks and shorten the path from a user's goal to its accomplishment.

"The biggest gap is, I believe, not having OpenMetadata in the infrastructure. I'll start with that. Yeah, I think not, I think as a data teams, we have more focused on or isolated problems. We touched upon earlier, right? So, you know, my as a, my world as a data engineer is about pipelines, you know, maybe Suresh has a data analyst problem, it's the dashboards, then there's business users and everything else. I don't think we are organizations are not there yet collectively thinking how to manage all of these things together and the outcome-based approach. I think where we are with OpenMetadata, I think that's what we are enabling to understand the landscape and enable the users to actually get the value out of it."

Srinivas identifies the primary gap in data management infrastructure as the absence of a comprehensive platform like OpenMetadata. He argues that data teams often focus on isolated problems, leading to a lack of collective thinking about managing the entire data landscape. He believes OpenMetadata's outcome-based approach is crucial for understanding this landscape and enabling users to derive true value from their data.