AI Code Review Transforms Development Through AST Context

Key Quotes

"understanding what is happening at a 1500 person engineering team is extremely challenging like my job was the head of product and you know in order to be good at my job i needed to have a really good understanding of what people were working on right i could tell you at any given point in time the most important priorities we should be working on but i had no idea whether we were allocating our engineering effort you know aligned to those priorities and even my engineering counterparts didn't know right like with that large of an organization um it's hard to know what people are doing"

Kayvon Beykpour explains that managing a large engineering team, even with clear priorities, makes it difficult to track actual engineering effort allocation. This highlights the challenge of maintaining visibility into project progress and resource alignment within massive organizations. The lack of clarity extends even to engineering leadership, indicating a systemic issue in understanding team activities.

"the problem that we were trying to solve with macroscopic is fundamentally like closing this gap we want to build we want to help bring instantaneous visibility to the people who need it right your ceos your ctos your product leaders your engineering leaders your tech leads -- they all want different levels of understanding different levels of granularity or coarseness of like what is happening what is the state of things -- and they want it quickly they want it accurately based on the actual ground truth and the ground truth for any software company is the code"

Kayvon Beykpour states that Macroscope aims to bridge the visibility gap for various stakeholders, from executives to tech leads. He emphasizes that the solution provides quick and accurate insights derived from the codebase, which he identifies as the ultimate ground truth for any software company. This underscores the product's core value proposition: delivering real-time, code-based understanding of project status.

"we've made a pretty big technical bet around leveraging the ast um in order to generate really useful references to set the llm up to be successful and what i mean by that is that we found that if you're just doing the simple thing of like ryan has a diff let's send the diff to the llm and just have it summarize it like that will generate a summary but we've found that um it is much more robust and accurate and often times magical to be able to set the llm up with a more comprehensive understanding not just of the change ryan made but the context of how the codebase around that change works"

Kayvon Beykpour describes Macroscope's technical approach, highlighting their reliance on Abstract Syntax Trees (ASTs) to provide context to Large Language Models (LLMs). He explains that simply sending a code diff to an LLM yields less robust results than providing it with a broader understanding of the surrounding codebase. This AST-driven context, he argues, leads to more accurate and insightful summaries of code changes.

"we found that if you're just doing the simple thing of like ryan has a diff let's send the diff to the llm and just have it summarize it like that will generate a summary but we've found that um it is much more robust and accurate and often times magical to be able to set the llm up with a more comprehensive understanding not just of the change ryan made but the context of how the codebase around that change works and so we leverage the ast to create a graph of the codebase and we don't just send ryan's diff to the llm we send you know what are the callers of this function that ryan changed what are the downstream functions the sort of like in references -- or the example usages of the function that ryan changed and by sort of supplying the llm with all of those things the diff the references -- it allows the llm to have a more coherent and robust summary of what that change was"

Kayvon Beykpour elaborates on Macroscope's use of ASTs by explaining that they provide the LLM with not only the code diff but also contextual information like function callers and downstream dependencies. He states that this comprehensive input, derived from an AST-generated graph of the codebase, enables the LLM to produce more coherent and robust summaries of code changes. This approach aims to go beyond basic summarization by offering deeper, context-aware analysis.

"our thesis is that -- you know humans should not be spending time reviewing prs for bugs like we ought to be able to delegate that as soon as possible like if the a proper you know well instrumented well built ai code review tool should be better at identifying bugs -- correctness issues that could cause you know problems in production better faster cheaper than a human would that is our belief"

Kayvon Beykpour presents the core thesis behind AI-powered code review: that humans should not be the primary reviewers for bugs. He believes that well-built AI tools can identify correctness issues more effectively, faster, and cheaper than humans. This perspective suggests a future where AI handles bug detection, freeing up human engineers for more complex tasks.

"my hot take would be anyone who's like opposed to these tools -- and thinks that they're a waste just hasn't used a good one right because we felt the same way like we used really bad ai code review tools and the vibe you get after using them is just like get this out of github this is a waste of time -- but once you've used a good ai code review tool it's a one way door you are not going to go back to not having an ai code review tool period"

Kayvon Beykpour shares a strong opinion on AI code review tools, asserting that opposition often stems from negative experiences with subpar tools. He argues that once a user experiences a high-quality AI code review tool, it becomes an indispensable part of their workflow. This "one-way door" analogy emphasizes the transformative potential of effective AI code review.