Moving AI From Proof of Concept to Production Requires MLOps

Key Quotes

"It's very easy to deploy an AI application if you ignore all the hard parts. If you're using OpenAI GPT-5, we can't even guarantee that today at 9 o'clock and at 12 o'clock it's exactly the same model. They may have changed something. You have no idea, and suddenly your system doesn't behave anymore."

Maria Vechtomova explains that deploying AI applications can appear simple by overlooking critical complexities. She highlights the unpredictability of even advanced models like GPT-5, where unseen updates can alter system behavior without notice. This underscores the need for robust MLOps practices beyond initial deployment.

"What excites me personally the most is that we deliver value, and that's the tricky part because everyone is just overly excited about the GenAI development and trying to apply it everywhere where it doesn't necessarily make sense. That part doesn't necessarily excite me, but I think the whole LLM hype did a lot of good for MLOps. Finally, we have attention to monitoring. That's a topic that was always important, but no one cared about it that much."

Maria Vechtomova expresses enthusiasm for AI applications that deliver tangible value, contrasting this with the trend of applying GenAI indiscriminately. She notes that the LLM hype has inadvertently increased focus on MLOps, particularly monitoring, a previously overlooked but crucial aspect of AI system management.

"From what I've seen, people mostly go the easy route, ignoring all of these hard parts. And they also detach all the pieces, right? Typically, you have some kind of data processing, and then you have PDFs. You are going to parse these PDFs, do OCR, do chunking, you use some vector search, you also maybe extract some metadata, store it in some SQL database, and then you define tools for your agent."

Maria Vechtomova observes that many individuals simplify AI system development by neglecting essential components. She describes a common approach where disparate elements like data processing, OCR, vector search, and database storage are handled separately, leading to a fragmented system architecture for agents.

"And then you have this is when you deploy the agent. And in I like Databricks a lot, and I use Databricks a lot. So there are processes that MLflow allows for. So for example, when you register a new version of the model, the deployment job will start, and it's not just it's going to deploy right away. Now it has an evaluation step using this kind of approach that I was talking about before. And then you have a manual approval step. So actually a human is going to look at it, maybe evaluate in certain ways, look at traces, maybe, and then we can go and deploy."

Maria Vechtomova details a structured deployment process, specifically mentioning Databricks and MLflow, where model registration triggers a deployment job that includes an evaluation step. She emphasizes the importance of a manual approval stage, where a human reviews the model's performance and traces before it is deployed to production.

"The funny thing is, when I started with AI within a product, and also I looked online, people were saying for software engineers, it's nothing new, right? Because it's just an API call away. And right now, if I hear you say like evaluations, figuring out your context, figuring out what actually is valid with regards to using an agent or a model behind the scenes, then the governance and the guardrails, if that all of a sudden becomes also responsibility of a software engineer, it's a lot on their plate because they don't just have to think of implementation, but also these are all data science principles."

Maria Vechtomova reflects on the evolving perception of AI integration for software engineers. She notes the initial misconception that it was merely an API call, contrasting it with the current reality where engineers must also manage evaluations, context, governance, and guardrails, which are essentially data science principles.

"And what I talk about as an MLOps principle a lot is traceability and reproducibility. You need to know what data was used, you need to know what code was used, what environment was used. And with ML models, it's more straightforward than, I guess, with the agents because, as I said, there are so many more moving pieces."

Maria Vechtomova highlights traceability and reproducibility as core MLOps principles, emphasizing the need to track data, code, and environment. She points out that achieving this is more challenging with agents due to their inherent complexity and numerous moving parts compared to traditional ML models.

"I don't think anyone knows. No, that's really the freshest. Well, how to do it well, I think it's hard because, I mean, now we need to think about security and all the attack vectors that can happen on your systems because if you're dealing with actual customer data, and some other customer can, by mistake or intent, get data from other customers, I don't know, try to get information about, I don't know, how much sales there are, things that they are not supposed to know, right?"

Maria Vechtomova addresses the significant challenges of deploying autonomous agents, particularly concerning security. She states that effectively securing systems that handle customer data against various attack vectors, including accidental or intentional data breaches, is an unsolved problem. This difficulty stems from the potential for agents to reveal sensitive information they are not meant to access.