AI Hype Obscures Non-Determinism Risks and Undermines Open Knowledge

Key Quotes

"A lot of folks know this I have friends who were you know teaching in these disciplines and and creating you know software in and and you know bayesian systems and and you know other adaptive systems for years and decades and so a thing that folks you know in the stack overflow communities will know is that they're nothing new under the sun right this this is something that has been around for many years and that certainly large language models and and the sort of related tech represent a breakthrough an evolution and there's lots of cool things that they can do but this is not something that comes out of nowhere or that there wasn't any prior art or that um coders particularly are saying oh my gosh we couldn't imagine that this was coming it is a um maybe a step change"

Anil Dash argues that the core concepts behind AI and machine learning are not new, having existed for decades. He points out that many in the software development community, particularly those familiar with Stack Overflow, recognize this history. Dash emphasizes that while large language models represent a significant evolution, they are built upon existing foundations and are not entirely novel inventions.

"And so yeah i i wanted to call it out because it's also um you know a couple parts one is you know it's bad for the economy it's bad for culture it it is bad because you have opportunists coming in but also as somebody who loves to make stuff and loves to see like the brilliant coders i know make stuff um you miss out like making stuff with code is awesome and it's like this great creative endeavor and we don't need the bs like we don't need to lie about it we don't need to make up that it like we don't need to exaggerate because the part that is real is cool and and so i i just end up very and you know not to be emotional about it but i am like i end up very resentful about these people who are misrepresenting it saying oh it it it does these exaggerated magical lies because the part that is real is cool and we could evaluate it more honestly these ways and also prepare against the harms right we could talk about what is broken or risky or dangerous more honestly if we if we just told the truth"

Anil Dash expresses frustration with the misrepresentation of AI technology, stating it is detrimental to the economy and culture by allowing opportunists to exploit exaggerated claims. He believes that the genuine capabilities of AI are already impressive and creative, and that lying or exaggerating about them detracts from this reality. Dash argues that an honest assessment would allow for better preparation against potential harms and a more accurate understanding of what is broken or risky.

"And the challenge with that is then a non expert person is putting the bugs in the wrong category but like as a technical person we can kind of translate that right we know why it's doing that but the effect of that is the non expert people are being exploited right you know because they're sort of attributing a certain degree of magic to it that the you know some of the people building these systems are taking advantage of right and and i just also think of like put all that like that's very theoretical really practically one of the magical things about software is it's deterministic like zeros and ones is a great thing right a falsifiable assertion is a great thing a test that passes is a great thing we all know that feeling of like i got this build to complete i got the single light up i got this you know this html page to render like whatever it is where you're like my code runs is an incredible feeling and part of the reason why is because it's deterministic behavior because of the nature of zeros and ones and the nature of llms is not that right and now the problem is we are trying to apply non deterministic systems to a lot of scenarios where you should have deterministic code like llms are bad at that thing and so why are we trying to use them as the hammer and all these things that ain't nails right"

Anil Dash explains that the anthropomorphic interface of large language models (LLMs) can lead non-experts to miscategorize errors, attributing them to human-like thinking rather than technical issues. He contrasts this with traditional software, which is deterministic and verifiable through tests. Dash highlights the danger of applying non-deterministic LLMs to tasks requiring predictable outcomes, likening it to using the wrong tool for the job.

"And so i i think 100 there's going to be somebody that starts as a prompt engineer who becomes a rust coder 100 somebody who goes down the rabbit hole yeah exactly so you have to have a gateway drug um so i have high confidence in that like people will always be curious people will always well the things will always break the abstractions will always leak and there's always going to be an error message where you're like what the hell is that and also one of the lessons of stack overflow coding is profoundly social people love hacking together they love showing off they love being proud of what they made they love interacting together knowledge is profoundly social i think and so i think that that aspect is always going to pull you further down the stack so i don't i don't worry for a minute about oh this is going to attenuate people's you know curiosity down down the stack at all"

Anil Dash expresses confidence that AI tools, like prompt engineering, will serve as gateways for people to explore deeper into coding, similar to how previous abstractions have functioned. He believes that curiosity, the inherent nature of technology to break or reveal complexities, and the social aspect of coding will naturally draw individuals further down the technical stack. Dash asserts that the human desire to build, share, and understand will prevent a decline in technical curiosity.

"And the reason that every single one of the big you know ai platforms and big llms is good at helping people code is because they were trained on stack overflow period yeah right and it's the generosity of this community in sharing information openly that made it possible and so there's i feel a social obligation to be reciprocal to like share that spirit and so when you know and so to to not you know return the favor and be generous in spirit and say no no no this is this is this rare thing that only these handful of people handful of companies can do and not listen to the expertise of the community who are saying treat this like other technologies and like there are some exceptions i'm sure there are some people that are like no no this is unlike anything else before and you have to treat it as a magical special thing but overwhelmingly 99 9 of coders i talk to of you know product people i talk to designers i talk to are all like can we just be normal about this"

Anil Dash emphasizes that the success of AI platforms in coding assistance is directly attributable to their training on Stack Overflow data. He feels a social obligation to reciprocate the community's generosity by advocating for a reciprocal spirit. Dash argues against the notion that AI is a rare, exclusive technology, asserting that the vast majority of technical professionals advocate for treating it as a normal technology, similar to others.

"And so you know the meta thing i would say the like the bigger thing i would say is um