Cognition Labs launched the demo video of its first ever fully automated AI software engineer named “Devin AI” on 12 March 2024, which has its code editor compiler as well as a browser to browse the stuff. Which blew up the whole internet and the developer’s community.
There was excitement and fear at the same time. Devin AI promised faster and more efficient solutions to the software engineering community. But how does the question of Devin AI being a scam rise? let’s find out!
Before diving into the details let’s quickly know What Devin AI is and what solutions it promises to the world.
What is Devin AI?
Devin is a super-intelligent program developed to assist software developers in software engineering tasks. Devin has its own compile, code editor and an independent browser to search for the stuff he doesn’t know on the internet.
Devin AI was made by the developers of Cognition Labs. They claimed Devin to be the first AI software engineer who could do various tasks related to software development independently.
Just after the trailer release of Devin, it got significant hype on all the social media platforms as a tool that can replace software engineers and take away their jobs. People were excited and feared at the same time. Before moving on to “Is Devin AI a scam?” let’s find out the capabilities of the Devin AI as promised by their developers.
Technical capabilities of Devin AI
According to Cognition Labs, Devin AI can learn how to use unfamiliar technologies, build and deploy end-to-end apps, find and fix bugs in codebases, train its own AI model, address bugs and feature requests in open source repositories, contribute to mature production repositories and solve real-world freelance problems relates to software engineering which they tested by giving Upwork problems to solve.
They tested Devin on the SWE benchmarks and as per the claims by the Cognition Labs, the model outperformed GPT-3.5, GPT-4, SWE-Llama 7B, SWE Llama 13B and Claude 2 by getting 13.86% issues resolved by their model.
Now you might wonder why even after these outstanding capabilities and performance Devin AI is being called a scam. Let’s find out the shady things hidden under this glamour.
Why is Devin AI a scam?
The dilemma of Devin being a scam or fake started from a video on a YouTube channel named ” Internet of Bugs” where he kind of exposed the Devin AI and put the real thing in front of the public.
In the 36-minute-long video on his channel, he explained how the developers of the Devin AI manipulated the benchmarks and the actual performance of the model.
The developers of Devin were accused of cherry-picking the problems and giving them to Devin to solve also the Upwork gig they solved was cherry-picked.
The person who posted the problem on Upwork himself said that Devin did not solve the problem as per the requirements.
The YouTuber also pointed out that the file visulisation.py on which Devin was working and trying to solve bugs was written by itself only.
After all these accusations on the model Cognition Labs is getting titled as fake over popular social media platforms including YouTube and Twitter. There is no official response from the CEO of Cognition Labs yet.
Link to the Mentioned YouTube video: Click here to watch