Treating LLMs as APIs Requires Defensive Programming and Structured Output

Key Quotes

"I do think that we are all tired of like every tech product out there is trying its best to find a way to put ai in the product like oh any any gmail any calendar app let's put ai in there and it gets a bit tiring like I wouldn't be too surprised if like fire hoses and refrigerators also come with ai features these days like it it feels a bit nuts."

Vincent Warmerdam expresses a common frustration with the pervasive integration of AI into everyday products. He highlights how the desire to incorporate AI can lead to its inclusion in unnecessary or even absurd applications, suggesting a saturation point where the novelty wears off and becomes tiring for consumers.

"My main way of dealing with it is more along the lines of like okay like this genie is not going to go back into the bottle in a way but if we're able to do more things with these LLMs if they're able to automate more boilerplate and that sort of thing then it seems that then I can try and get better at natural intelligence as well instead of this artificial intelligence."

Warmerdam proposes a proactive approach to the integration of LLMs, viewing them as tools that can automate tedious tasks. He suggests that by offloading boilerplate work to AI, individuals can focus on enhancing their own "natural intelligence" and pursue more complex or creative endeavors, rather than simply accepting AI as a replacement for human thought.

"The one thing that's very annoying is you can write down x is equal to 10 and then have all sorts of cells below it that depend on x you can then delete the cell and the notebook will not complain to you about it even though if anyone else tries to rerun the notebook x is gone it won't run and your notebook is broken and you can't share the thing anymore."

Warmerdam points out a significant drawback of traditional notebook environments like Jupyter. He explains that the lack of explicit dependency tracking can lead to broken notebooks when cells are deleted or reordered, making them unreliable and difficult to share, a problem that newer tools aim to address.

"The main thing I had with the course was like I would an LLM is a building block at some point but it's very unlike a normal building block when you're dealing with code because normally with code you put something into a function like one thing comes out but in this particular case you put a thing into a function you have no idea of front what's going to come out and not only that but you put the same thing in twice and something else might come out as well."

Warmerdam describes the unique challenge of integrating LLMs into software development. He contrasts LLMs with traditional code functions, emphasizing their inherent unpredictability and non-deterministic nature, which requires a more defensive and cautious approach to their implementation.

"One of the things you always want to do is like think a little bit about caching and there's a python library called disk cache that I've always loved to use and I highly recommend people have a look at it."

Warmerdam advocates for the use of caching mechanisms when working with LLMs, specifically recommending the diskcache Python library. He explains that caching is crucial due to the cost and computational expense of LLM calls, ensuring that repeated identical requests are not processed unnecessarily.

"The main thing that's always nice and the and when you're dealing with LLMs is you always want to be able to say in hindsight like okay how did this LLM compare to that one you want to compare outputs and then just writing a little bit of a decorator on top of a function is the way to put it in the sql light and you're just done with that concern."

Warmerdam highlights the utility of diskcache for comparing LLM outputs. He suggests that by using a decorator, developers can easily store and retrieve LLM responses in a SQLite database, enabling straightforward comparison and analysis of different models or prompts after the fact.

"The main thing that Simon Willison's library does allow you to do is you are able to say well I have a prompt over here but the output that I'm supposed to get out well that has to be a JSON object of the following type and then you can use Pydantic."

Warmerdam discusses the llm library by Simon Willison, emphasizing its capability to enforce structured output formats. He explains that by combining this library with Pydantic, developers can specify that the LLM's response must conform to a particular JSON schema, ensuring more predictable and usable data.

"The main thing you get is I remember when the first open AI models sort of started coming out I worked back at at Explosion we made this tool called spaCy there and one of the things that we were really bummed out about was LLMs can only really produce text so this structural like if you wanted to take a substring in a text for example which is the thing that spaCy is really good at then you really want to guarantee well I'm going to select a substring that actually did appear in the text."

Warmerdam reflects on the limitations of early LLMs, contrasting them with NLP tools like spaCy. He notes the frustration that LLMs primarily output text, making it difficult to guarantee structured information extraction, a problem that has been a significant focus for improvement in the field.