New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Fully agentic without human decision is just dumb idea. And blame infra provider in the X post is dumb too, HA/FT should be done by company itself, not by provider even if provider have that feature.
Most recent offsite backup was 3 months old? I guess that's the AI's fault also.
"Oops, you're right, I accidentally ran rm -rf *. What would you like me to do next?"
We've contacted legal counsel.
Presumably because PocketOS is about to be sued out of existance.
I notice the PocketOS web site has a railway error on its front page...
What's up with Railway btw? I keep seeing their name pop up.
This has happened multiple times now.
Why do we continue to not learn the lesson?
Why were prod and staging on the same infra? I don't know anything about Railway, but in Wndows/Azure terms, prod and dev should always be on a different domain. The stars of stupidity had to align in so many ways for the AI to be able to do this. These people used worst practices every step of the way, and now they're trying to throw all of their vendors under the bus.
Not ideal on Railway's part, but also a token that powerful shouldn't be left lying around by the user for AI to discover and use. AI should be in its own sandboxed environment and have the harness program running either in a VM or as a completely locked down separate user.
Their backup strategy likely failed as they don't factor in growth or doesn't value good backup strategy
Even article written with llm!
Somehow, I just remember what stallman said "There user control the software, or software control the user" even though it different context.
People lately using "tools" beyond what it must be the "tools"
That's a huge wall of text to basically show no sense of accountability whatsoever.
For anyone without a Twitter account: https://archive.ph/T3LU6
This is an expected behavior if you don’t add “make no mistakes” into your workflow
Meanwhile, legal counsel:
They are a fly-by-night clone of Vercel/ now.sh much like Glitch. COVID money lead to a lot of them springing up offering a lot of low-cost container compute before AI consumed it all. I guess the reason their name keeps coming uyp is because they are as vibe-coded as the customers they host.
You always need to have a human in the loop, a proper human accountability-sink is an absolute necessity for when things go wrong.
Also, letting your AI assistant crawl random websites can have serious consequences, a growing number of websites are deploying attacks that target AI systems browsing the web.
A number of websites are attempting to vandalize the machines of anyone using AI assistants
Yes - in theory. But in real life, and certainly in this case, the story IMO is presented upside down.
Taken from Railway's web site, the home page:
So, it's not just a case of "Claude made a hiccup" albeit a desasterous one, nope this in fact is a case of an idiot using a complete AI stack getting killed pretty much. Because only an idiot would bet and build his whole company on AI. Because only an idiot would bet and build his whole company on vibecoded and in the cloud". And because the idiot still didn't get it, as is clearly indicated by his explanation of the two possible reasons Railway still didn't provide a clear statement. The true reason is (a) Railway is based on the same utter idiocy, and (b) hence can not possibly know with clarity what happened, why it could happen, and how it could happen. Because for that they'd need to truly and fully know and understand what their shit AI built, which, if they could they wouldn't have built their crap around AI in the first place.
It's simple: None of the idiots have a clue how and why AI SHTF.
The only ones I feel pity for is the customers of the idiot company.
So, "Our" in title is not you?
anything AI produce, doesnt belong in production without proper security and review from human expert first.
I wish its me.
The interesting piece is how Railway app is architected.
Not denying that AI can do shit, but if there were more checks and balances on Railway for keeping Backups tied to the transactional volume
Without any clear understanding.
IMHO, its like I have backups. Yes, in a backup folder in the same server
Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue
https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
Its sad to see the Providers getting name shamed here when it seems the error lies deeply within the chain.
telling the agent to create a backup & verify is just one prompt away, it's bullshit those "people" are blaming on others when they're not sentient enough to realize this step.
Yeah, that's the crux of the matter. Sometimes I wonder whether we are really Sapiens or just Automatons, left to their own devices, most humans would revert to the path of least (mental) effort regardless of the risks and when shit happens: "it wasn't supposed to do that".
NORMALLY it is not supposed to happen, but we are not living in a predictable world, at least not to the point we think we do, everything is built on such precarious fundamentals, that not only the wars or AI could destroy civilization, but also a simple glitch could poof the whole Universe out of existence. The path we are on is an extremely low probability one, keeping the entropy out is the full time job of existing and abandoning thinking is a sure-fire way of fading out slowly or rapidly, depending on luck.
Btw, sad to see that tomshardware has been turned into an ad spam orgy.
Premise error. You are assuming that the agent does as told, which however is not the case here.
Also, If they really were sentient they wouldn't have an "agent" (read: ai bot) in the game in the first place.
make no mistake!
Never keep your backups in the same server is a bit like "Never write your PIN directly on your debit card."
and be careful! (frequently omitted but very important too)