LMArena's AI Evaluation North Star: Integrity, Real-World Feedback, and Vertical Expansion

Key Quotes

"The way the company started was as an incubation by Anj [Midha]. So what he did is he kind of like found us at Berkeley and picked us out of the basement and was like, 'Hey, these guys seem like they're onto something' and started working with us really early. Gave us, you know, gave us some grants. He was not, you know, a16z was not the only one to do this; we also had a great grant from Sequoia, but Anj was in particular quite, quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even were committed to starting a business."

This quote highlights the foundational support Arena received from Anj Midha, who incubated the project before it was officially a company. Anastasios Angelopoulos explains that this early backing, including grants from venture capital firms like a16z and Sequoia, provided crucial resources for development, demonstrating a unique approach to startup genesis.

"It became clear that the only way to scale what we were building was to build a company out of it. That the world really needed something like Arena, Arena being really a place to sort of measure, understand, and advance the frontier AI capabilities on real-world users, on real-world usage, based on organic feedback. And that in order to achieve the scale and, you know, distribution necessary and the quality of course of the platform necessary to do this effectively, we would need to start a company out of it."

Anastasios Angelopoulos articulates the strategic decision to spin Arena out as a company, emphasizing that an academic or nonprofit structure would not allow for the necessary scale and quality. He explains that the mission to measure and advance AI capabilities through organic user feedback required the resources and operational framework of a commercial enterprise.

"The purpose of money at a company is to give you cards to flip. It's to say, 'Hey, you have enough resources necessary and so that if your first bet fails, you can make another bet and another bet.' Of course, so that's not to say that we're going to spend all of it. Of course, you want to spend things responsibly. Having said that, the platform is actually quite expensive to run. We fund all of the inference on the platform."

Anastasios Angelopoulos describes the strategic role of capital in a company, framing funding not as a budget to be depleted, but as a set of opportunities for strategic investments. He clarifies that while responsible spending is key, the significant operational costs, particularly for inference, necessitate substantial financial resources to maintain the platform's free usage for millions of conversations.

"The leaderboard delusion is a paper that critiques LMArena. The main, pretty like, yeah, brutally, well, you know, I would say unscientifically. And let's be clear, Cohere wasn't doing that well on LMArena. Cohere was like 74th. It's all good, you know, it's actually not, it's a respectable place that they had on the leaderboard. I don't even think it was really Cohere people, like the Cohere model developers doing this, it was more their research side. But in any case, what does the, what does the leaderboard delusion say? It says that LMArena was, what their claim is, that LMArena was doing this undisclosed, quote unquote, private testing on our platform."

Anastasios Angelopoulos summarizes the core claims of the "Leaderboard Delusion" paper, which critiqued Arena's evaluation methods. He explains that the paper alleged undisclosed private testing of pre-release models, which the paper argued created inequities. Angelopoulos notes that Cohere's own performance on the leaderboard was respectable, suggesting the critique may have stemmed from their research division rather than direct model developers.

"In reality, as you probably know, we've been doing this pre-release testing for a long time. Our community loves it. They love basically getting, like, secret codenames, yeah, the secret codenames like Nano Banana, yeah, all that. So Nano Banana, by the way, started on us, right? And people loved it. It went like global sensation."

Anastasios Angelopoulos defends Arena's practice of pre-release testing, highlighting its popularity within the community. He explains that users enjoy the excitement of early access to unreleased models, often identified by secret codenames like "Nano Banana," which can generate significant public interest and even global sensation, as seen with that specific example.

"The platform integrity comes first. To the platform, the basically the public leaderboard that we show on LMArena, I think of as a charity. It's a loss leader for us. We don't really make money on the public leaderboard. You can't pay to get on the public leaderboard. It's not like a Gartner in that sense. It's not like any of these, you know, pay-to-play systems. Never going to be like that. Models are going to be listed on the leaderboard whether or not the providers pay and whether or not they're getting a good score. They can't pay to take it off either."

Anastasios Angelopoulos emphasizes Arena's commitment to platform integrity, characterizing the public leaderboard as a charitable endeavor rather than a revenue-generating service. He states that models cannot pay to be listed, removed, or influence their scores, ensuring that the leaderboard remains an unbiased reflection of real-world performance based on millions of user votes.

"Every user is earned. You have to earn them every single day. They can leave at any moment. They're fickle. And so all the time you have to be thinking about, 'How do I provide this person value? Learning how are they using my website? What more could I give them?' And how do I build in all the retention mechanisms so that they stay and then they're also bringing their friends?"

Anastasios Angelopoulos discusses the critical nature of consumer retention, explaining that users must be continuously provided value to remain engaged. He stresses that users are "fickle" and can leave at any time, necessitating a constant focus on understanding user behavior, enhancing the platform's offerings, and implementing retention strategies to foster loyalty and organic growth.