AI Transforms Data Engineering: New Assets, Testing, and Uptime Demands

Key Quotes

"The introduction of generative AI and AI Engineering to the technical ecosystem is changing the scope of responsibilities for data engineers and other data practitioners. Of note is the fact that: AI models can be used to process unstructured data sources into structured data assets, AI applications require new types of data assets, the SLAs for data assets related to AI serving are different from BI/warehouse use cases, the technology stacks for AI applications aren't necessarily the same as for analytical data pipelines, because everything is so new there is not a lot of prior art, and the prior art that does exist isn't necessarily easy to find because of differences in terminology."

Tobias Macey explains that the advent of generative AI is fundamentally altering the role of data engineers. He highlights several key shifts, including the ability of AI to transform unstructured data, the emergence of new data asset requirements for AI applications, and the altered service level agreements (SLAs) for AI-related data compared to traditional business intelligence use cases. Macey also notes the challenges posed by new technology stacks and the lack of standardized terminology in this rapidly evolving field.

"Because everything is so new there is not a lot of prior art, and the prior art that does exist isn't necessarily easy to find because of differences in terminology."

Tobias Macey points out that the novelty of AI engineering means there is a scarcity of established practices and documentation. He further elaborates that even when existing knowledge is available, it can be difficult to access or understand due to the inconsistent use of terminology across different projects and teams. This lack of a common language and established methods creates a significant hurdle for engineers entering this space.

"Another aspect of the ways that these language models and generative systems have changed the work of data engineers is that we have new types of data assets that we're responsible for with vector databases and vectors being the biggest example where before we would have tabular data you would maybe do some feature engineering on that and then send that to the training and maybe hydration of a machine learning model for being able to create some prediction or perform some action but now we need to take structured semi structured and unstructured data turn them into those vector embeddings and then store them in a relatively new technology for many in the form of these vector databases so that the models can retrieve that at inference time and be able to do it quickly and effectively."

Tobias Macey describes how generative AI introduces new data asset types, specifically mentioning vector databases. He contrasts this with previous practices involving tabular data for machine learning models. Macey explains that data engineers now need to convert various data formats into vector embeddings and store them in vector databases for efficient retrieval during AI model inference. This represents a significant evolution in data structuring and management for AI applications.

"Unless you were working in an organization that relies heavily on real time streaming data the sla or service level agreement or the reliance of that data and its timeliness has changed a lot and the uptime in particular of that data has changed a lot where if you were building something for a data warehouse that was feeding into a business intelligence system there's a pretty high probability that if it goes down for 15 minutes or an hour in the middle of the night well you reload the data warehouse or update data or fix some bugs it's not going to be that big of a deal but if you are running a vector store and it is powering a customer facing llm that is doing inference and providing an interactive use case if that same downtime happens it's a much bigger deal."

Tobias Macey highlights the increased criticality of data timeliness and uptime due to AI applications. He contrasts the impact of downtime for traditional business intelligence systems with that of customer-facing AI models, such as those using vector stores for LLM inference. Macey emphasizes that for interactive AI use cases, even short periods of downtime are far more consequential, necessitating higher reliability standards for data infrastructure.

"Maybe one of the biggest new requirements especially for data engineers is the fact that the way that we test these different workflows has to change where we have things like data quality monitoring we have unit tests we have integration tests to make sure that if the logic breaks if some new data comes in that breaks our prior assumptions we would react to it and fix it but as the models start to become more of the core execution either in terms of the actual pipelines themselves or in terms of the serving of what the data is feeding into everybody in the whole workflow needs to be able to include experimentation and evaluation as that means of building confidence and verifying changes and maintaining functionality."

Tobias Macey argues that the testing methodologies for data workflows must evolve with the integration of AI models. He explains that traditional methods like data quality monitoring, unit tests, and integration tests are insufficient when models become central to execution. Macey stresses the necessity of incorporating experimentation and evaluation into the workflow to build confidence, verify changes, and ensure functionality in AI-driven systems.