Rethinking AI Benchmarks for Human-Centric Usability and Safety

Key Quotes

"Is a car that wins a Formula 1 race the best choice for your morning commute? Probably not. In this sponsored deep dive with Prolific, we explore why the same logic applies to Artificial Intelligence. While models are currently shattering records on technical exams, they often fail the most important test of all: the human experience."

Andrew Gordon and Nora Petrova introduce a core problem in AI evaluation: technical benchmarks, like those used to rate Formula 1 cars, do not necessarily translate to practical, everyday usefulness for humans. They argue that AI models, despite high scores on exams, may not be suitable for real-world human interaction.

"Most reporting on benchmarks is done on technical benchmarks these days right and that is where you get the model you give it a set of evaluations maybe on one theme or maybe an exam and then you get a score and humans aren't really involved in that loop."

Andrew Gordon explains that current AI benchmarking often relies on technical evaluations, where models are given tests and receive scores without direct human involvement in the process. This approach, Gordon notes, overlooks the crucial element of human experience in assessing AI performance.

"People are increasingly using these models for very sensitive topics and questions for mental health for how should they should navigate problems in their lives and there is no oversight on that and in any other area where these topics are discussed there is a lot of regulation and then a lot of ethical conduct built into it whereas here is kind of the wild west at the moment and some companies are taking more seriously than others and trying to study the ways in which humans are uh using the models for for more personal topics and and problems."

Nora Petrova highlights a significant gap in AI evaluation: the lack of oversight and regulation when models are used for sensitive personal matters like mental health advice. Petrova contrasts this with other fields that have established ethical guidelines and regulations, describing the current AI landscape for such applications as a "wild west."

"I think for me there's three big areas I think where where we've sought to improve first of all as you mentioned sample so obviously the sample for that chatbot arena is anybody right we don't know anything about them we don't collect any demographic data so they are just people going there anonymously prompting the models and giving their preference data now obviously that's great you get a huge amount of data which is fantastic but you know nothing about the people giving the data which is fairly suboptimal."

Andrew Gordon critiques existing human preference leaderboards, like Chatbot Arena, for their lack of demographic data on participants. Gordon explains that while these platforms gather large amounts of preference data, the absence of information about the users makes the data "suboptimal" for understanding diverse human experiences.

"Then in terms of specificity any for anybody that's used chatbot arena all you're doing is saying i like this response more or i like this response more right in the real world that kind of data is is useless in a sense right it gives you a nice way to make a nice leaderboard of ai models but it tells the companies nothing about why that preference has been has been given."

Andrew Gordon further elaborates on the limitations of current human preference evaluations, specifically noting the lack of specificity in user feedback. Gordon points out that simply stating a preference for one response over another, as done on platforms like Chatbot Arena, provides little actionable insight for AI developers about why a preference exists.

"Also the model doesn't know their background and culture so it's very hard to align with them but the other possibility is that potentially models are just not very good at that and that would be potentially an effect of the data they've been trained on right because we know very little i mean obviously models are trained on the entire internet but when you train a model on the entire internet do you get a personality that really represents what people want."

Nora Petrova discusses potential reasons for AI models underperforming on metrics like personality and cultural understanding. Petrova suggests that this could stem from the models' inability to account for individual backgrounds and cultures, or it could be an inherent limitation of their training data, which is derived from the broad internet.