Interesting. That's an (expired) Cloudflare certificate, and those are Cloudflare IP addresses I'm being sent to.
I wonder if Cloudflare broke this somehow (whoops) or if Thunderbird themselves screwed up here. I don't see a CAA record that would tell DigiCert they can't issue this either.
But that's a much bigger problem that it might appear.
If Cloudflare gets right to the wire and then is like "Oh, DigiCert couldn't issue for whatever reason" then that's a process problem at Cloudflare not DigiCert.
A reasonable strategy is to renew say, 30 calendar days in advance. But that would mean Cloudflare (or at least, some automated system that nobody was looking at) knew 30 days ago this would happen and nothing was done about it.
The Cloudflare certificate which expired was for *.thunderbird.net which you will be able to find in crt.sh for yourself - and was issued a year ago under Cloudflare's intermediate†
None of those R3 certificate expire today, and the expired cert I was originally served when this was a fresh news item was from that Cloudflare intermediate CA under DigiCert.
It's possible that normally Thunderbird has Cloudflare trust their backends via the Let's Encrypt certificate rather than using Cloudflare's private "origin" CA for that purpose. That would give them the flexibility to cut Cloudflare out of the loop if necessary. But what exactly happened here I expect we'll learn in a post-mortem.
† I assume but don't know for certain that in fact DigiCert operates this intermediate CA simply on behalf of Cloudflare but entirely on their own premises and with them ensuring its obeys the CA/B BRs and other trust store policies. I think the idea of letting somebody else operate unconstrained intermediates with them doing all the work died out after it sank Symantec.
This usually happens when the company does not centrally manage their certificates.
Some guy creates a certificate, leaves the company, there is some renewal mechanism which just works, nobody knows anymore where it exactly is and then it starts failing.
> Some guy creates a certificate, leaves the company, there is some renewal mechanism which just works, nobody knows anymore where it exactly is and then it starts failing.
Which is why part of the standard operating procedure (SOP) of adding any new service is to add it to your monitoring infrastructure. This includes certificate expiration even if it is supposed to be automated.
Automation breaks sometimes, and you have to know when it does breaks so a 'manual override' can be done.
Right. Certificates are actually one of the easier monitoring problems - you can build an affirmative monitoring setup, you can get them off the shelf, it's very much a solved problem.
Contrast stuff like: Did we pay the utility bills? Oh, you thought Jim did it, Jim thought Sarah did it, Sarah thought you did it, and so they weren't paid, and this morning bright and early an engineer from the utility company disconnected our supply, we are dead in the water. Don't worry, your utility company is probably a local monopoly and so they definitely have excellent customer support /s
I never understood why people were so upset at Mozilla's CEO earning a couple million per year in salary. Seems like a rather reasonable rate for running a medium size company that doesn't give out stock options/RCUs
I suspect it's at least partially the 'giving a raise/increase while also firing people at the same time' optics which doesn't sit well with folks.
It doesn't actually strike me as 'reasonable rate' for a company which is losing market share with their main products, has a history of multiple failed product/service launches, and is having to fire people to cut costs. I (and many others) could get the same results for a mere $700k/year.
Hell, I'll almost certainly fail, but I've at least got some ideas to try out to (sloooowly) turn the ship around, for that rate. "Keep on truckin'" clearly isn't working.
> It doesn't actually strike me as 'reasonable rate' [...] I (and many others) could get the same results for a mere $700k/year.
Surely, you can see the flaw in this logic though? If you set the salary too low, then only unqualified people will apply, and you'll be guaranteed to get bad results. You have to pay a competitive salary if you want even a chance of getting a skilled leader. Worse, it is harder turn around a company that is doing poorly than to cruise along at the helm of a well performing one. Which means that you both want to have higher standards when selecting a CEO of a poorly performing company, and also have to pay a premium to attract qualified candidates at all.
Except she is the one who has been overseeing this slide into irrelevancy for many years now....for whatever reason the Board, instead of replacing her, rewards her.
A different way to look at it is that the CEO provides >$2M in value to the company, whereas many of the engineers were not providing their $100k+ salary of value.
It's because Mozilla is visibly, painfully, repeatedly, and incomprehensibly, grossly mismanaged.
They have an enormous budget, but are funded from basically a single source, and the future of that funding is not guaranteed.
They have dramatically dwindling market share. This is critical because their voice in the future of the web could easily be drowned out by the interests of Google-Apple-Microsoft.
Their non-browser projects are inconsistently conceived, and unreliably maintained.
These situations have been created and compounded under the direction of their CEO.
They have no roadmap to fix these issues, any of which could be fatal.
Mozilla is our best hope for the future of the free web, and their gross mismanagement makes me very fearful.
Every single year, they are funded well enough to remake the organization, and every single year they do not.
I say this as a loyal Netscape/Mozilla/Phoenix/Firebird/Firefox user since always.
I say this with a continued belief that Brendan Eich created a situation (or a situation was created around Brendan Eich, if you prefer) where he could not have effectively led Mozilla.
And I say this with awareness that 99% of us would probably fail to lead Mozilla (it is a delicate balancing act) -- but that only the current management has dipositively done so.
If there was any accountability to users or shareholders, the Mozilla executive team would have been replaced years ago. It is difficult to watch this astonishing waste of resources, and massive dereliction of duty to the public.
But who should she be replaced by? It's not like Mozilla just cherry-picked her over a sea of other amazing, willing-to-be-underpaid candidates who were all clamouring for the job. Being a CEO during tough times sucks.
I think Eich was actually lucky, in a twisted way. It's not like he managed to turn their fortunes around while he was their CTO for so long. He wasn't some powerless underdog who would have gained special superpowers as their CEO.
Even if he had avoided controversy to become their CEO, he would almost certainly have just been the one we're scapegoating now instead of Baker. She was our angle when Mozilla was founded and when Microsoft needed an antitrust spanking, and now she's uor devil.
I don't know who should be Mozilla CEO. But I know that the current management is failing in unexciting and repetitive ways.
Someone with new ideas should be Mozilla CEO. Someone who believes in the mission of the organization, and will use the organization's vast resources to chart a new path that respects that mission.
An executive search would have no trouble finding quaified candidates at that pay level. They might not work out either. But failing in the same way, over and over, is silly magnified to horrifying due to the importance of Mozilla to the web and the public.
I don't entirely agree about Eich. He was respected in the org, and obviously has ideas that exceed Mozilla's charter. A CTO is limited in their strategic scope, so I don't think his years working for Baker indicate he would fail in the same way she has.
There is nothing repetitive about the approaches of the last few Mozilla CEOs. They did not simply steer the same course as their predecessors. I'm not even sure where this idea could come from?
I'm also confused as to why you think an executive search would surely find a better candidate when it clearly hasn't. We can't just pretend the problem away. If we can't find a better candidate than the woman who ran MoCo when it was founded, then we're grasping at straws.
It also doesn't matter whether we want to believe Eich could have done better. He didn't, and he isn't going to be their CEO now. So I'd rather not pick at old wounds too much.
You picked first, with that “lucky… twisted” line. I was CTO without reports until I took SVP Engineering at start of January, 2013. Until then, I had influence but no line of management authority over engineers or others except my great EA who started in 2012. Nevertheless, I was the key principal engineer / chief architect and later main exec backing everything from Firefox with add-ons instead of the suite approach (Netscape, SeaMonkey), Thunderbird, Mozilla spinning out of AOL, us starting HTML5 with Apple and Opera via the WHATWG, Mozilla rejoining ECMA, Rust, Servo, TraceMonkey, Firefox OS, Mozilla Research, and long term JS leadership that culminated in ES6.
The difficulties Mozilla has had in finding a better direction have a lot to do with depending too much on search revenue, Google now and from 2004 except for the ill-fated Yahoo deal in end of 2014 which blew up in 2017. At Brave, I’ve had to find other revenue than a Google Search deal. I couldn’t have done it at Mozilla, but never mind me: Mozilla can’t find other revenue either, and this probably dooms them as Firefox sheds users while Google pays on traffic (I’m told).
Those are arguments that Mozilla should find a different CEO, not that paying a lower salary to their CEO would improve outcomes. The part I don't get is way people are so viscerally appalled by Mozilla allocating a fairly normal level of compensation for their CEO (relative to the size of the company)
Right, but the argument is that the CEO person and salary are mismatched, I think -- not necessarily that the salary is inappropriate.
A CEO needs to earn their keep. Baker does not, and somehow keeps getting increases while there are zero metrics on which the organization is improving.
Not complaining at all, but why should a for-profit, wholly-owned subsidiary of a nonprofit pay 10x what I make at Brave? Mitchell’s answer was because she could make $3M elsewhere. True or not, that doesn’t settle the matter unless Mitchell is the only possible CEO ever. Perhaps she is, in the scenario where Mozilla has to collapse back into a nonprofit due to Firefox losing too much share. She could sell Firefox to Private Equity, as a prior CEO wanted to do. These are the hard decisions that only a long term leader might make, and sell to enough folks in the community. Nothing looks easy from here. But the minimum price of a CEO is not $3M/year, I can tell you that!
A CEO's pay is often a function of the organization's budget.
Mozilla has a huge budget, so it's not terribly surprising that some well-paid person is in charge of it.
(Somehow Mozilla's CEO's pay has managed to increase while the organization's budget has been decreasing. This is inexplicable and unjustified.)
Regardless of how well- or over-paid Mozilla's CEO is, though, she is not adding comensurate value to the organization, and I invite her to earn $3MM elsewhere starting tomorrow.
OTOH, if tomorrow she announces The Mozilla Fund, and reveals that they've been accumulating excess budget (tens, hundreds of $MM/yr) into a permanent trust that will support perpetual development of Firefox and related technologies at a secure withdrawal rate -- and that all has been done legally from an accounting and tax perspective, and that we've all just overlooked it in the annual reports ... then I nominate her for Chairperson Emeritus who can keep her stupid salary, and hire a clueful product person as CEO.
I know little about SSL certificates. Could someone tell me why they should expire regularly? Is it for precaution? Is it just because "good practices". What's the difference (from a security point of view) between a certificate that expires in 3 months and one that expires in 3 years?
There are two related but different things going on here.
1. Revocation is broken. The obvious way for revocation to work (your browser checks if the certificates it sees have been revoked) not only has negative privacy implications (so e.g. Firefox does not implement this) it also plain doesn't work. It has "fail-open" behaviour for a bunch of sad but inevitable social reasons. In theory we have viable fixes for this, but in practice we can fix really big problems (e.g. a whole CA is compromised) with a lot of effort and smaller problems (e.g. your IT guy emailed your private keys to a scammer) mostly not at all. Thus, expiry is the hard stop, if your certificate expires in six weeks, the scammer loses any benefit [edited] in six weeks. Problem "solved" or at least mitigated.
2. The Web PKI can only evolve at the rate permitted by certificate expiry. Anything faster will set important stuff on fire, because no matter how hard you try you can't publicise changes enough to ensure people are aware of them. e.g. Everybody needed to stop using SHA-1 certificates, that took several years, when the same happened for MD5 it took so long to be effective that even though it was deprecated before a reliable collision was known there were multiple successful attacks anyway. Under Apple's current 12 month rule, if we need to fix something ASAP, we can either break the world, or we wait until late 2022. But under the historic Five year rule that becomes break the world or don't fix it until 2027. That's a pretty big difference in how agile we can be without telling everybody, "Sorry, we broke the whole Internet, reinstall and try again".
In the sense that you, as the owner of a name, can choose to have your certificate say it must always be presented with OCSP stapling, and thereby your certificate can have revocation that works without privacy concerns?
Yes.
But for the larger global problem, OCSP stapling is a solution in the same sense that "Don't dig up any more coal, oil or gas" is a solution. Yes, I urge you to do this in both cases. It's a good idea. But, is everybody going to do it tomorrow? Next year? In my lifetime?
Sure, however this has the effect of showing any eavesdropper exactly which certificates you're trusting (and when), and thus in most cases which web sites you're visiting. It also tells the Certificate Authorities the same things about sites they've issued certificates to.
Now, I assume that the list of people who visit Porn Hub isn't valuable enough to justify the CA (DigiCert again) explicitly monitoring who visits the site and selling it when they already charge PornHub good money for a certificate. But who knows?
OCSP is default enabled in Firefox actually, so you're already doing that.
Setting required to true just makes it actually demand a response instead of allowing an MITM to block it making OCSP ineffective.
At the end of the day we need to start progressively rejecting certificates that aren't OCSP stapled to fix both sides of this, but for now this is the best fix available for technical people who understand how OCSP works.
Even if your certificate expires after 3 years, the certificate renewal process should be automated. The long validity of 3 years leads many people to not consider this necessary.
If the certificate is only valid for three months however, many people will automate the renewal right away, because nobody wants to do this manually every couple weeks.
From a security POV, shorter lifetimes require more periodic checks for the server's identity. E. g. a Letsencrypt-issued certificate using the ACME protocol will validate the server really belongs to the given domain more often, which is a nice property I think.
The longer a certificate is valid for, the longer an attacker has to try compromise it somehow. Relying on certificate revocation instead of short-lived certificates is not an ideal solution. As systems have improved such that regular renewal can be easily done without human intervention (and if done properly, with no less, or in fact possibly more, assurance of security) it makes sense to take advantage of this by reducing certificate lifetimes. Like single-use credit cards, to give an analogy that a non-techie man-on-the-street can latch on to.
edit: With 3 year certificates there is also a greater risk for certificates expiring unexpectedly because the renewal process happens so infrequently. The person with the knowledge on the renewal process might not be at the company anymore by the time the certificate expires.
I don't know for the rest, but for letsencrypt you don't even have an option to set the expiration date, as far as I know. You have to make sure certificates generated there are regularly refreshed (every three months). So it might well be an imposition from the Certificate Authority.
I wonder if Cloudflare broke this somehow (whoops) or if Thunderbird themselves screwed up here. I don't see a CAA record that would tell DigiCert they can't issue this either.