AI Accelerates Automation by Quantifying Economic Value and Reshaping Expertise

Key Quotes

"when one of the ai labs wants to teach their models how to be better at poetry we'll find some of the best poets in the world that can help to measure success via creating evals and examples of how the model should behave and one of the reasons that we're able to pay so well to attract the best talent is that when we have these phenomenal poets that teach the models how to do things once they're then able to apply those skills and that knowledge across billions of users hence allowing us to pay 150 an hour for some of the best poets in the world"

Brendan Foody explains that Mercor hires top poets to train AI models in poetry. The high hourly rate is justified because these poets' expertise can be applied across a vast user base, making the investment scalable and profitable. This highlights Mercor's model of leveraging specialized human talent for AI development.

"the largest disconnect that we were seeing in ai research is that everyone was focused on academic evals like gpqa for phd level reasoning or imo for olympiad math which were wholly disconnected from the outcomes that customers actually care about of how do we get the model to automate a medical diagnosis or a legal draft or preparing a certain financial analysis of a company"

Brendan Foody points out a critical gap in AI research: a focus on theoretical benchmarks rather than practical, real-world applications. Foody's company, Mercor, aims to bridge this by using experts in fields like medicine, law, and finance to develop evaluations that reflect actual customer needs and economic value.

"the largest takeaway is the rate of model improvement at economically valuable tasks is incredible like if you look at the level that gpt4 scored on this model right a frontier model a year ago and that against gpt5 today the delta is profound and so it often gets can you put a number on that or somehow yeah call it a 25 30 improvement per year per year exactly"

Brendan Foody shares a key finding from their research: AI models are improving at an astonishing rate in tasks that have economic value. Foody quantifies this improvement at approximately 25-30% per year, indicating a rapid acceleration in AI's capability to automate complex professional work.

"the economic value is in a way still at zero well so it's interesting i think what you're getting at is there's sort of two key things the models struggle at that humans tend to be very good at the first is these longer horizon tasks of not just something that we could do in a few hours but something that might take us 50 or 100 hours to do and then the second thing is integrating multiple tools with our response and going about doing these things maybe interacting with people as one of those elements"

Brendan Foody identifies two significant limitations of current AI models: their difficulty with long-term, multi-stage tasks and their inability to seamlessly integrate multiple tools or interact with humans. Foody suggests these are areas where human expertise remains crucial, though he anticipates rapid advancements in these capabilities within the next six to twelve months.

"the best analog is a rubric if you have some a rubric for how to grade a rubric for how to grade so if you have here's like if the poem has you know evokes this idea that is inevitably going to come up in this prompt or is a characteristic of a really good response will you know reward the model a certain amount if it says this thing will penalize the model if it styles the response in this way we'll reward it those are the types of things in many ways very similar to the way that a professor might create a rubric to grade an essay or a poem"

Brendan Foody explains that the most valuable data for improving AI, particularly in subjective domains like poetry, is a well-defined rubric. This rubric acts as a grading system, outlining specific characteristics of good responses and providing clear criteria for AI to learn from, analogous to how a professor grades an essay.

"my firm belief is especially for economically valuable tasks we'll move towards a world where people do things once instead of the investment banker redundantly analyzing a data room to prepare an analysis of a company you know every couple of weeks for a new project or a new customer they'll teach the model how to do that once in the particular domains that they operate in and similar to building software once they'll be able to use that many times as they use their agent instead of the customer support rep monotonously responding to tickets every day and they'll find the mistake that the agent makes they'll turn that into an rl environment and then all of a sudden the agent will be able to solve that problem many times"

Brendan Foody predicts that economically valuable tasks will increasingly shift towards a model where humans teach AI systems once, and then those systems can perform the task repeatedly. This is akin to software development, where a fixed cost is invested upfront to create a reusable asset, thereby transforming knowledge work into a more efficient, agent-based system.