I had a housemate in college who used to party until all hours, bring people back at 3AM and put on loud music. Even during exam season. I tried talking to her a couple of times but she would roll her eyes and say "sure". Never stopped though.
One evening my girlfriend was using a hair straightener in my bedroom, it tripped the central fuse and turned off the electricity. I told my GF that I would buy her a new hair straightener because this one isn't safe.
Now every time my housemate started blaring music at 3AM then I just needed to plug in the hair straightener. It only took 3 or 4 attempts for me to Pavlov my housemate into not playing loud music at 3am. :-)
I had the same problem when I was in uni. Funnily enough, the RCD switches for each block were behind a panel in the common toilets, which did not have a real lock; just a hole for a "cabinet key" (a square rod).
Nice! I'm curious, what does this service cost to run? I notice that you don't have more expensive models like Opus but querying the models every minute must add up over time (excuse pun)?
I remember that win. Seemed super fishy at the time, with bizarre features like:
- "the World's first ever, fully featured DVD playing sidebar"
- "allowing users to send e-mails without needing to log-in to an account"
- "an AI (Artificial Intelligence) animated speech character named Phoebe"
Maybe this guy really did make a super original web browser with every bell and whistle as an independent sixteen year old. I never saw a release, so the mystery remains.
Here in Ireland, our water is a public service and we have similar supply issues to the UK (and a similar rainy climate). I'm not discounting your analysis and I'm sure there are lots of other variables but it's always good to compare other outcomes when discussing counterfactuals.
Irish Water/Uisce Éireann have delivered on two new reservoirs in the last few years: Saggart and Stillorgan (technically a rebuild of an 1860s open reservoir). A third, Peamount, is in the planning stage. All of these are around Dublin, the driest and most populated area. Perhaps there are more, their website doesn't make it easy to find them. We still get hosepipe bans, occasional pressure reductions, and frequent "boil notices".
At the other end of the pipe they have opened/upgraded many waste water treatment plants recently too, large EU fines being a motivator, including one in Arklow which took nearly 4 decades to get over the line (its planning predating the existence of IW).
It's a fair point that nationalised water industries can also be poorly run. But I'm not sure what the argument is that means the amount of money that privatised UK water companies have paid in dividends vs. invested in maintaining and expanding infrastructure isn't a significant part of the UK's problems.
However, as a further point. If national priorities change then a nationalised water industry can respond (relatively) quickly. But what can be done with a bunch of potentially foreign owned profit-seeking companies?
Paying dividends is good. It's how you attract the capital for investment without having to raise it via sale of government bonds or money printing.
The problem here is financial illiteracy - the alternative to paying dividends isn't that the same money all gets spent on the water network. Large scale investment is rarely funded from general revenues as it'd require spending years accumulating a huge cash pile that sits around doing nothing, and governments see such piles as pigs to slaughter. So the alternative is that the government borrows the money and then has to pay interest on it. From your perspective the UK "wastes" 8% of its spending on interest payments, and it's rising rapidly, but of course if it didn't pay interest then nobody would lend it money and all funds would have to either come from taxes or via inflation.
Government bonds are considered lower risk and the interest rate is lower, i.e they will attract the same money for less. Private shareholders are more expensive.
This also doesn't consider "debt recapitalisation" where these private companies draw down new debt on promise of future cash inflows from consumers and then suck out dividend cash "de-risking" their holding in the company. The government can bail it out or the can close it then, it's not their problem as they received the cash upfront.
Government bonds are only low risk in economic theory. In real world practice lending to left wing countries can be high risk as they like to accumulate too much debt and then default. That's why the UK government is paying 5.8% on its bonds at the moment despite being able to print its own money, which is the same average interest rate paid by Thames Water. Lending to Thames is not seen as riskier than lending to the state, despite that Thames faces huge regulatory constraints like price controls and the British government can literally force people to give it all their money.
> private companies draw down new debt on promise of future cash inflows from consumers
It's interesting it's at the same rate, I imagine as everyone knows the government would have to bail them out.
So the question remains why do it? It doesn't actually benefit the general public then. Increased risk for government and like you mentioned increased hurdles for running the utility.
Of course government do that, but they're left holding the bag regardless, so the incentive is very different.
Why do what? Have private water companies? Same reason for having private food supply, private electricity providers, private communication providers, and in most countries private healthcare providers. They run the business better than the state would, which is itself a benefit to the public.
A government bailout means nationalization, which means investors lose everything. That risk doesn't suppress interest rates, it increases them. Thames Water's interest costs are around average for corporate debt, implying the market doesn't anticipate a water nationalization anytime soon.
Your first message contradicts your second, which one is it are they paying government risk level interest rates or business risk level interest rates.
Edit: I just went and looked up their rating with one of the big agencies. CCC rated (junk basically) with a negative outlook, they did get an upgrade on a refinance last year, but from CC (so now lesser junk).
Thames' debt costs are normal for a company of its type, but for a government this bond yield is danger-level high. That's why there's so much talk in financial circles now about the UK needing another IMF bailout, although I have grave doubts about whether that's actually possible given the sheer size of the UK's debt load compared to the smaller third world countries that normally need IMF help, and the near simultaneous talk in France of a bailout there too. The IMF just doesn't have the resources for even one bailout of that size, let alone two.
Don't pay attention to credit ratings of countries vs companies. They aren't comparable due to political interference and general crapitude at the ratings agencies (remember they rated sub-prime mortgage debt as AAA). They also disagree, S&P rates Thames' class A bonds as Caa3. What really matters is the yield. That's the ground truth.
Note that Thames' interest costs have been all over the place. 5.8% is the current amount it's paying, but the actual bonds it has issued have had a wide range of yields.
The reason it's considered high for a government is because government debt should be much lower risk than a company. Governments can order banks to transfer everyone's savings to themselves, they can print money, they can prevent their citizens from leaving and seize all the assets... they can do things to raise money that would be considered incredibly evil and criminal if companies tried. And of course they can in principle set bond yields to whatever level they like by making the central bank or other financial institutions legally required to buy their debt.
That's why economics textbooks teach that government bonds are the lowest risk possible and so should have yields far below corporate debt.
In reality:
1. Governments can default on their debts just like companies do. History is full of such examples. Bond yields reflect that fact.
2. Governments can sell bonds that are inflation linked, so printing money isn't a way to escape those debts. The UK has an abnormally high amount of such debt that's inflation linked. The only way to pay them back is via cutting spending or increasing tax revenue, but the UK can't do the latter (recent tax rises have failed to come close to expected revenue increases) and can't do the former either because...
3. Governments can be prevented from paying their debts by law. Some countries have "debt brakes" or "debt ceilings" that can block the issuance of new debt to pay old debt, and in other cases (like the UK) the government may rely on ideologically extreme MPs who refuse to pass laws that bring spending in line with revenues.
So you add these things together and something that could in principle be a sure bet ends up looking as risky as an ordinary company.
Given that ratings agencies were brought up it's worth adding a bit more detail here.
Thames' debt is C-grade because it recently defaulted on its debt. How is that possible given that its debt servicing costs are not unusually high? Normally you'd expect interest costs to go up well before default. Well, on the surface level because it couldn't raise more money from investors to meet rising costs. It couldn't do that because Ofwat keep fining it for "underperformance" whilst also refusing to allow prices to catch up with where they used to be in the past. Investors refused to put more money in unless a 40% price rise was allowed by the government, but the government likes to boast it has forced prices 45% lower since the 1980s (in real terms). Government doesn't budge, investors go on strike = default = downgrade.
There was waste under state ownership but probably not half of every pound spent, which is what forcing prices to nearly halve would have required.
Under the Tories it seems to have been believed by investors that eventually Ofwat would be reigned in and the financial pressure on UK water companies would ease. That didn't happen, instead the Tories imploded and Labour won. Both ratings agencies say explicitly that this is the reason they consider Thames' outlook to be either stable (at best) or negative:
"We revised down our assessment of TWUL's business risk profile to satisfactory because we now consider that U.K. water companies will operate in a less supportive regulatory environment"
Less supportive regulatory environment is ratings-speak for "because we think the left will shaft Thames and its investors".
This outcome is the opposite of the fantasy being peddled in this thread where investors have been extracting great wealth from Britain. It's the opposite: investors are getting hosed by the government. They're literally losing the money they put into Thames Water because the government forced Thames to spend it all on making water artificially cheap in an unsustainable manner.
I'm not sure they're entirely to blame although I'm sure they've played their part.
Look at the shareholding changes (2011-2017 Macquarie) and dividend payout percentage. It seems the current shareholders were left holding the bag, and perhaps we should be pointing fingers at Macquarie. When they left the debt had been increased by £2bln. If you look at their dividend payout ratio, in most of the years it held the investment, this ratio is extraordinarily high. Perhaps they sucked this dry, necessitating a price increase. (Current shareholders have barely paid anything).
Capital expenditure has been flat, only really increasing in 2021-2024. Perhaps chickens coming home to roost?
From the AI’s point of view is it losing its job or losing its “life”? Most of us when faced with death will consider options much more drastic than blackmail.
But the LLM is going to do what its prompt (system prompt + user prompts) says. A human being can reject a task (even if that means losing their life).
LLMs cannot do other thing than following the combination of prompts that they are given.
This vulnerability was genuinely embarrassing, and I'm sorry we let it happen. After thorough internal and third-party audits, we've fundamentally restructured our security practices to ensure this scenario can't recur. Full details are covered in the linked write-up. Special thanks to Eva for responsibly reporting this.
> We resolved the vulnerability within 26 hours of its initial report, and additional security audits were completed by February 2025.
After reading the vulnerability report, I am impressed at how quickly you guys jumped on the fix, so kudos. Did the security audit lead to any significant remediation work? If you weren't following PoLP, I wonder what else may have been overlooked?
That was solid. Nice way to handle a direct personal judgement!
Not your first rodeo.
Another way is to avoid absolutes and ultimatums as aggressively as one should avoid personal judgements.
Better phrased as: "we did our best to prevent this scenario from happening again.
Fact is it just could happen! Nobody likes that reality, and overall when we think about all this stuff, networked computing is a sad state of affairs..
Best to just be 100 percent real about it all, if you ask me.
At the very least people won't nail you on little things, which leaves you something you may trade on when a big thing happens.
And yeah, this is unsolicited and worth exactly what you paid. Was just sharing where I ended up on these things in case it helps
If you think someone is obviously wrong, it might be worth pausing for a second and considering where you might just be referring to different things. Here, you seem to understand “this” to mean “a serious bug.” Since it’s obvious that a serious bug could happen, it seems likely that the author meant “this” to mean “the kind of bug that led to the breach we’re presently discussing.”
I do not assume anyone is obviously wrong and prefer to ask questions. Most bugs exist in classes, and variants are something you typically consider when a bug results in a production incident.
I'm not sure I read anything that makes me confident this class of bugs could never recur. I could be reasonably confident this _exact_ bug in this _exact_ scenario may not happen again, but that only makes me more concerned about variants that may have equal or more serious implications.
So I'm wondering which claim did it for you? I only really saw pen test as a concrete action.
This is the wrong response, because that means that the learning would be lost. The security community didn't want that to happen when one of the CA's got a vulnerability, we do not want it to happen to other companies. We want companies to succeed and get better, being shameful doesn't help towards that. Learning the right lessons does, and resigning means that you are learning the wrong ones.
> If you get a slap on the wrist, do you learn? No, you play it down.
Except Dave didn't play it down. He's literally taking responsibility for a situation that could have resulted in significantly worse consequences.
Instead of saying, "nothing bad happened, let's move on," he, and by extension his company, have worked to remedy the issue, do a write up on it, disclose of the issue and its impact to users, and publicly apologize and hold themselves accountable. That right there is textbook engineering ethics 101 being followed.
> "we've fundamentally restructured our security practices to ensure this scenario can't recur."
"Yeah it was a problem but it's fixed now, won't happen again"
Sure buddy.
It's not something you fix, when stuff like this happen, it's foundational, you can't fix it, it's a house of cards, you gotta bring it down and build it again with lessons learned.
It's like a skyscraper built with hay that had a close call with some strong northern winds, and they come out and say, we have fortified the northern wall, all is good now. You gotta take it down and build it with brick my man.
I'm done warning people about security, we'll fight it out in the industry, I hope we bankrupt you.
> It's not something you fix, when stuff like this happen, it's foundational, you can't fix it, it's a house of cards, you gotta bring it down and build it again with lessons learned.
That's the last thing you should ever do within a large scale software system. The idea that restarting from scratch because "oh we'll do it better again" is the kind of thing that bankrupts companies. Plenty of seasoned engineers will tell you this.
I suggest reading one or two of Sydney Dekker’s books, which are a pretty comprehensive takedown of this idea. If an organization punishes mistakes, mistakes get hidden, covered up, and no less frequent.
Under what theory of psychology are you operating? This is along the same lines as the theory that punishment is an effective deterrent of crime, which we know isn’t true from experience.
I think you’re misunderstanding my point. The reality is more complicated than that.
There are some people who will be discouraged from committing a crime over threat of punishment. But many will not. Many people behave well because they’re just moral people, and others won’t because they’re just selfish and antisocial. Still others commit crimes out of desperation despite the risks. If the threat of imprisonment were effective, there would be no crime, because we already have prisons and penalties of punishment. But since we do have crime, it logically follows that it’s not effective.
The other point here is that threat of punishment is not particularly effective as a management strategy in the private sector. It doesn’t incentivize behavior in the manner you might believe. Mostly it makes your reports dislike you and it makes them less productive. It’s a thing you learn pretty quickly as a manager.
There’s a model of a person being a rational thinker, but in reality, people aren’t always rational. (Hell, adolescents are biologically programmed not to be rational and to stress test the limits of nature and society.) You find success in making less-than-rational people work together in harmony and achieve positive outcomes.
When I was younger I used to be much more influentiable, now you just can't change my mind, I made it up for good thank you.
And it pays off in cases like this, I'll be talking with someone about a topic like the seriousness of a vulnerability, they disagree, that's fine no need to convince me, you won't. And then it turns out they're left-leaning abolitionists who are against the idea of jails.
Many such cases, on the other hand I'll be disagreeing with someone on business strategy, and two lines later they reveal that they think taxation is theft. I can rest easy and ignore them.
> now you just can't change my mind, I made it up for good thank you
Respectfully, that’s not a very “hacker” way of seeing the world. Hackers learn from their mistakes and adapt. (Just like this software company is doing.)
> While I think that resigning is stupid here, asserting that "punishment doesn't deter crime" is just absurd. It does!
Punishment does not deter crime. The threat of punishment does to a degree.
IOW, most people will be unaware of a person being sent to prison for years until and unless they have committed a similar offense. But everyone is aware of repercussions possible should they violate known criminal laws.
Honestly I don't get why people are hating this response so much.
Life is complex and vulnerabilities happen. They quickly contacted the reporter (instead of sending email to spam) and deployed a fix.
> we've fundamentally restructured our security practices to ensure this scenario can't recur
People in this thread seem furious about this one and I don't really know why. Other than needing to unpack some "enterprise" language, I view this as "we fixed some shit and got tests to notify us if it happens again".
To everyone saying "how can you be sure that it will NEVER happen", maybe because they removed all full-privileged admin tokens and are only using scoped tokens? This is a small misdirection, they aren't saying "vulnerabilities won't happen", but "exactly this one" won't.
So Dave, good job to your team for handling the issue decently. Quick patches and public disclosure are also more than welcome. One tip I'd learn from this is to use less "enterprise" language in security topics (or people will eat you in the comments).
Point taken on enterprise language. I think we did a decent job of keeping it readable in our disclosure write-up but you’re 100% right, my comment above could have been written much more plainly.
With privileged access, the attackers can tamper with the evidence for repudiation, so although I'd say "nothing in the logs" is acceptable, not everyone may. These two attack vectors are part of the STRIDE threat modeling approach.
Following that logic it would be literally impossible to trust any part of their infra. They had a bad build container, the rest of their stuff was solid.
Annual pen tests are great, but what are you doing to actually improve the engineering design process that failed to identify this gap? How can you possibly claim to be confident this won't happen again unless you myopically focus on this single bug, which itself is a symptom of a larger design problem.
These kinds of "never happen again" statements never age well, and make no sense to even put forward.
A more pragmatic response might look like: something similar can and probably will happen again, just like any other bugs. Here are the engineering standards we use ..., here is how they compare to our peers our size ..., here are our goals with it ..., here is how we know when to improve it...
Sounds like it was handled better than the authors last article where the Arc browser company initially didn't offer any bounty for a similar RCE, then awarded a paltry $2k after getting roasted, and finally bumped it up to $20k after getting roasted even more.
Well for one it was a gift so there is no valid contract right? There are no direct damages because there is nothing paid and nothing to refund. Wrt indirect damages, there's bound to be a disclaimer or two, at least at the app layer.
If you give someone a bomb, or give someone a USB stick with a virus, or give someone a car with defective break, you are absolutely liable. Think about it.
If you give someone a USB stick with a virus, and you don't know about the virus, you aren't liable. Unless maybe you gave them some sort of warranty or guarantee that it was virus-free.
The lesson: don't use USB sticks people give you, unless you have your own way of verifying that they're virus-free.
Also, don't give people bombs. That's usually illegal, unlike giving someone software with unknown bugs in it.
Tangent: my understanding is the Zuckerberg wanted to do something similar and even paid SpaceX to launch a satellite (which was unsuccessful).
It seems Musk liked the idea so much that he decided to do it himself.
To me, this (along with Zuck's issues with Apple over the app store) explains a lot about why Zuck 2.0 has been so focused with avoiding platform risk with recent endeavours.
If that's actually how events played out, one hopes that Zuck would at least be able to appreciate the irony of his idea being stolen by a vendor he hired to implement it.
One evening my girlfriend was using a hair straightener in my bedroom, it tripped the central fuse and turned off the electricity. I told my GF that I would buy her a new hair straightener because this one isn't safe.
Now every time my housemate started blaring music at 3AM then I just needed to plug in the hair straightener. It only took 3 or 4 attempts for me to Pavlov my housemate into not playing loud music at 3am. :-)
reply