Bridging Prototype and Production--Data Science Coding Practices

Key Quotes

"I saw the pain of machine learning engineer so my manager he's a very good data scientist but he's not very good at coding so a lot of time he would the machine learning engineer needs to like rewrite a lot of his code so I want to like you know how do I write the code in a way that when I give it to him he can immediately like just minimal edit and then use it for production that's one thing another thing I found is it's a pain for me to you know write a lot of code in a notebook and then later try to understand what I wrote and also you know if I run my notebook in different orders then it will show different output so it's very messy at the same time it's not reusable I cannot it's really hard for me to reuse it for a different project that use similar pieces of code like how do I extract it out so that I can reuse different components and another thing that goes with it is you know you need testing right how do I make sure that this code works for different circumstances so the book is about kind of good Python practices you can say for data scientists including writing Python variables Python classes Python functions like when should you even use Python classes because sometimes you know you don't it's not really necessary for data science projects also you need testing also like hard coding right like a lot of time like in Python projects in a data science project you see a lot of global variables being used hard coded variables and um I offer the alternative which is using a configuration so you have a configuration file where you put all the values so you can very easily change it right but at the same time you have your all code logic in a different place so that it looks very clean and also you can unit test independent of the hard coded values."

Khuyen Tran explains that the motivation for her book, "Production Ready Data Science," stems from her experience as a data scientist in a startup. She observed the difficulties machine learning engineers faced when trying to use code written by data scientists, often requiring significant rewrites. Tran also highlights the personal challenges of managing code in notebooks, including issues with reproducibility, understanding past work, and code reusability. The book aims to provide data scientists with good Python practices to create code that is immediately usable by engineers and is more organized and testable.

"I really like uv because of the fact that it have two files one is the lock files that you know that lock the dependencies which is like if you wanted to reproduce the exact dependencies you can do it but at the same time it had a pi project com with more flexible dependencies so yeah that's become that's why I really enjoy uv is also very fast yeah yeah the lock files would solve that say potentially pandas issue you're mentioning before yeah now you know you're going to get the exact version yeah which which is good yeah and also I would say not the problem and I I wrote it in the book but like with pip right like compare uv with pip for example a lot of the time you know when you run pip install something and then like there's a dependency issue but it's like install anyway the dependency conflict that is not like something to be ignored and if there's like there could be like potential downstream issues that come with like different like you don't match dependencies that the package required but with uv like if that happens it's gonna stop you right so you can try to find a way to match the dependencies of the packages that you use together you feel like it it presents you with information that you can kind of work off of compared to pip yes in that sense okay yeah yeah yeah"

Khuyen Tran expresses a strong preference for the tool uv as a dependency manager. She highlights that uv uses both lock files for exact dependency reproduction and pyproject.toml for more flexible dependency management, contributing to its speed and reliability. Tran contrasts this with pip, noting that pip can sometimes install packages despite dependency conflicts, which can lead to downstream issues. uv, on the other hand, is described as stopping the process when conflicts arise, enabling users to resolve them and ensure better compatibility between packages.

"Marimo is solved that exact problems right and oh also another thing Jupiter notebook is json based and it's like oh Marimo it allows you to write it as a notebook but then at the same time it's a dot py file like that is amazing like you have the best of both worlds the interactivity and you have a python script and if it needs a python script under the hood then you can do whatever you want with it just like a python script you can do unit testing very easily you you can just import it the functions or whatever from that notebook right like the dot py notebook yes exactly good practice also like another thing I really like Marimo like if you change you know like in notebook if you change something in one cell you you need to remember like which one depend on that one so that it of course you can run everything but you probably don't want to run everything if your notebook is very heavy so you run the dependent cells and to do it manually it's very difficult right to remember that but with Marimo it automatically run the dependent cells I also like the fact that it's like keep track of which variable like the dependencies between cells and oh another thing is it integrated very well with uv right so like if you have uv and you have Marimo it's like very nice yeah because one problem with notebook is you would like pip install something right I see people do that like in the notebook like pip install something yeah and it might even do it like deeper into the notebook like in cell 25 or something like that you know and they don't really specify the version of the package right or like there's no what what I try to say is it says like it says let's say you pip install pandas and then later I don't know one year later maybe you run that again with pip install pandas but it will be a different pandas version right if it installed from scratch so there are dependencies issues that would a notebook but with Marimo what happened is you don't even do pip install so let's say you import something like you let's say you import plotly right and then it will detect it and you say oh hey you're missing plotly dependencies like you want to install it and then when you install it it is tracked so what happens is the next time it's a track in like the top of the notebook which is a py file so the next time you can do it it used with uv so uv I don't remember the exact command but like uv run something something but like it reproduce exactly because you know it the code stay the same if the dependencies are exactly the same you will have the same output as you've shared these techniques with other people what has been the reaction of it like what have people thought of you know when you you you do a lot of writing you get a lot of comments on your your site and maybe even back with your other data science comrades or colleagues what would they say about Marimo and the changes there they I think they like it it does take some time like when I heard I told them about the features like oh this is amazing I want