Shopify's Sidekick: Embedding Human Opinion to Elevate AI Development

Key Quotes

"I did engineering in university here in Waterloo in Canada. I've always loved building things and so that was a very natural post-secondary education for me to have. From there, I did a couple of startups. I think this is back in the time before startups were that cool, so it was not that in vogue to do startups. It was a little bit, I felt like a little bit of a lone wolf."

Vanessa Lee explains her early career path, highlighting a passion for building and an independent spirit during a less popular era for startups. This demonstrates her foundational drive and willingness to pursue unconventional paths.

"We've just grown and grown and grown and become truly the operating system of merchants' businesses. I've worked on quite a few parts of our platform, online store and Liquid, some of the Horizon updates that you chatted with Glenn about. I had worked with the team on quite a lot, so it has been a firehose over the last six months, but perhaps one that I had already in some places dabbled in."

Vanessa Lee describes Shopify's evolution into a comprehensive business operating system, reflecting on her own broad experience across various platform components. This illustrates the expansive scope of Shopify's offerings and her deep involvement in its development.

"So the last couple of years, we've been working a lot on Sidekick. When we kind of came out earlier this year with a new architecture of Sidekick, we started seeing Sidekick be a lot more successful in most conversations. So I purposely kind of waited and held back the team from talking and shouting too much about Sidekick until I thought that it really drove some value."

Vanessa Lee details the development and strategic rollout of Shopify's AI assistant, Sidekick, emphasizing a deliberate approach to launching only when the product demonstrated tangible value. This shows her product management philosophy of prioritizing proven utility over early promotion.

"So one of the ways that we did that, we put a lot of work into the foundations of evaluations, but you also make sure that you have enough variety in that judge's training set where you also have essentially negative cases and grading it negatively so that the judge is also able to spot when it answered and tried to sell you a car, which we absolutely do not want."

Vanessa Lee explains a crucial aspect of AI development: the meticulous creation of evaluation datasets, including negative examples, to train AI judges. This highlights the rigorous, detail-oriented work required to ensure AI models behave as intended and avoid undesirable outputs.

"So if you think about how do we make sure that people are actually, people like on Shopify's side, like our opinions of how it should be, how it should act, those are all actually embodied in the ground truth set. And so that's kind of how you go from human, like it's not just LLMs making LLMs. We used LLMs creatively to help us scale. But for example, I would have a team say, okay, go and generate a bunch of conversations between Sidekick and, let's say, an LLM. And you take those conversations, you just edit them. And so that is the human in the loop."

Vanessa Lee clarifies the role of human oversight in AI development, explaining how human opinions are embedded into ground truth sets, which then guide LLM-generated data. This illustrates a hybrid approach where AI assists in scaling data creation, but human judgment remains central to defining desired AI behavior.

"And so one of the things that we, we did actually starting a couple years ago was use LLMs to start to properly categorize products and properly create attributes. So this is where I'm super proud of one of these launches. We've kind of worked on it behind the scenes over the years, but last year we actually basically embedded these predictions into Shopify."

Vanessa Lee discusses the application of LLMs to product categorization and attribute creation within Shopify, emphasizing the long-term development and eventual integration of these AI-driven predictions. This showcases how AI is being used to standardize and enhance product data across the platform.