>> was it the unions or executives that decided to offshore manufacturing?
>Neither. It was consumers, who prefer lower prices.
Right, because every executive who pursued offshore manufacturing was thinking, "gosh, how can I deliver even lower prices and better value to my customers?", and not "OK, we've shown the market will pay $X for product Y, how can I cut my costs and free up more money for bonuses and cocaine?"
Graphs of price indices (aside from a few sectors such as electronics where it was the core technology that improved, not labor efficiency) and wages over the last 50 years clearly show that the bulk of any offshoring savings were not passed along to consumers or front-line workers.
> Right, because every executive who pursued offshore manufacturing was thinking, "gosh, how can I deliver even lower prices and better value to my customers?", and not "OK, we've shown the market will pay $X for product Y, how can I cut my costs and free up more money for bonuses and cocaine?"
Suppose it used to cost you $1 to make something in the US that you had been selling for $1.50. The cost of domestic real estate and other things goes up, so now to make it in the US it costs you $1.60. If you sell for $1.50 you're losing money and to have your previous gross margin percentage you'd have to sell for $2.40. Meanwhile it still costs $1 to make it in China and one of your competitors is doing that and still selling it for $1.50.
Your options are a) don't raise prices, lose $0.10/unit and go out of business, b) raise prices, lose all of your sales to the company who still charges the old price and go out of business, or c) offshore manufacturing.
The only way out of this is to get the domestic costs back down, which is a matter of changing the government policy over zoning rules etc.
> Graphs of price indices (aside from a few sectors such as electronics where it was the core technology that improved, not labor efficiency) and wages over the last 50 years clearly show that the bulk of any offshoring savings were not passed along to consumers or front-line workers.
Are these graphs being adjusted for the increasing cost of things like domestic real estate having to be incorporated into the prices of everything? Even if you make it in another country you still have to pay for a domestic warehouse or retail store.
That's actually a terrific example, because the company already had a working, field-tested manually controlled version of their product, but they wanted a computer-controlled version to relieve radiologists of all of the drudgery and opportunities for error in the manual setup. So the design spec was to "faithfully replicate the manual setup procedures in a microcontroller," which was achieved. The problem was that the UI was utter dogshit, the training materials were poor (you can't have Marketing say "hey this computer handles all the details for you!" and then expect customers to pore page-by-page through the reference manual), and most clinics weren't really enforcing or QAing the operator training anyway. And since "the computer has a handle on it," the radiologists and senior techs who were supposed to be second-checking everything basically abdicated that responsibility, driven in part I'm sure by management that expected them to now have more "free hours" for other activities that Make Line Go Up. And nobody really raised a flag that this seemed troubling until multiple children died, and at the end of the day a lot of people could say "well, it wasn't my fault, I did what I was told to do" and sort of believe it.
If anyone doesn't think all of that is going to happen again on a massive scale as companies embrace GenAI, HAHAHAHAHAHAHA. AI will change a lot of things, but it won't fix human nature.
Modern CPUs do out-of-order execution, which means they need to identify and resolve register sharing dependencies between instructions. This turns the notional linear model of random-access registers into a DAG in practice, where different instructions that might be in flight at once actually read from or write to different "versions" of a named register. Additionally, pretty much every modern CPU uses a register renaming scheme, where the register file at microarchitecture level is larger than that described in the software-level architecture reference, i.e. one instruction's "r7" has no relationship at all to another's r7".
Caches aren't quite as mix-and-match, but they can still internally manage different temporal versions of a cache line, as well as (hopefully) mask the fact that a write to DRAM from one core isn't an atomic operation instantly visible to all other cores.
Realistically, the compiler is building a DAG called SSA; and then the CPU builds a DAG to do out of order execution, so at a fine grain -- the basic block -- it seems to me that the immutable way of thinking about things is actually closer to the hardware.
That doesn't affect what I said though. Register renaming and pipelining does not make mutation go away and doesn't allow you to work on multiple things "at once" through the same pointer.
It's still logically the same thing with these optimizations, obviously -- since they aren't supposed to change the logic.
Of course you should offer some method to disable caching/compression/encryption/ECC/etc. in intermediate layers whenever those are non-zero cost and might be duplicated at application level... that's the ancient end-to-end argument.
But that method doesn't necessarily have to be "something like O_DIRECT", which turns into a composition/complexity nightmare all for the sake of preserving the traditional open()/write()/close() interface. If you're really that concerned about performance, it's probably better to use an API that reflects the OS-level view of your data, as Linus pointed out in this ancient (2002!) thread:
Or, as noted in the 2007 thread that someone else linked above, at least posix_fadvise() lets the user specify a definite extent for the uncached region, which is invaluable information for the block and FS layers but not usually communicated at the time of open().
I think it's quite reasonable to consider the real problem to be the user code that after 20 years hasn't managed to migrate to something more sophisticated than open(O_DIRECT), rather than Linux's ability to handle every single cache invalidation corner case in every possible composition of block device wrappers. It really is a poorly-thought-out API from the OS implementor's perspective, even if at first seemingly simple and welcoming to an unsophisticated user.
O_DIRECT isn't about bypassing the kernel for the sake of reducing overhead. The gains would be small if that was the only reason.
O_DIRECT is used to disable cache replacement algorithms entirely in contexts where their NP-hardness becomes unavoidably pathological. You can't fix "fundamentally broken algorithm" with more knobs.
The canonical solution for workloads that break cache replacement is to dynamically rewrite the workload execution schedule in realtime at a very granular level. A prerequisite for this when storage is involved is to have perfect visibility and control over what is in memory, what is on disk, and any inflight I/O operations. The execution sequencing and I/O schedule are intertwined to the point of being essentially the same bit of code. For things like database systems this provides qualitative integer factor throughput improvements for many workloads, so very much worth the effort.
Without O_DIRECT, Linux will demonstrably destroy the performance of the carefully orchestrated schedule by obliviously running it through cache replacement algorithms in an attempt to be helpful. More practically, O_DIRECT also gives you fast, efficient visibility over the state of all storage the process is working with, which you need regardless.
Even if Linux handed strict explicit control of the page cache to the database process it doesn't solve the problem. Rewriting the execution schedule requires running algorithms across the internal page cache metadata. In modern systems this may be done 100 million times per second in userspace. You aren't gatekeeping analysis of that metadata with a syscall. The way Linux organizes and manages this metadata couldn't support that operation rate regardless.
Linux still needs to work well for processes that are well-served by normal cache replacement algorithms. O_DIRECT is perfectly adequate for disabling cache replacement algorithms in contexts where no one should be using cache replacement algorithms.
Finding defects is a good thing, and fixing defects is a good thing. Adding new features can be a good thing as long as it doesn't introduce or uncover too many new defects in previously stable code. But what is lacking in your development process that you keep finding "critical" defects that could affect a large number of users during the merge window?
It seems like bcachefs would benefit from parallel development spirals, where unproven code is guarded by experimental flags and recommended only for users who are prepared to monitor and apply patches outside of the main kernel release cycle, while the default configuration of the mainline version goes through a more thorough test and validation process and more rarely sees "surprise" breakage.
It certainly appears that Linus can't tell the difference between your new feature introductions and your maintenance fixes, and that should trigger some self-reflection. He clearly couldn't let all of the thousands of different kernel components operate in the style you seem to prefer for bacachefs.
> It seems like bcachefs would benefit from parallel development spirals, where unproven code is guarded by experimental flags and recommended only for users who are prepared to monitor and apply patches outside of the main kernel release cycle, while the default configuration of the mainline version goes through a more thorough test and validation process and more rarely sees "surprise" breakage.
Maybe if you're a distance observer who isn't otherwise participating in the project
What's the saying about too many cooks in the kitchen?
These concerns aren't coming from the people who are actually using it. They're coming from people who are accustomed to the old ways of doing things and have no idea what working closely with modern test infrastructure is like.
There wasn't anything unusual about the out-of-merge-window patches I was sending Linus except for volume, which is exactly what you'd expect in a rapidly stabilizing filesystem. If anything I've been more conservative about what I send that other subsystems.
> It certainly appears that Linus can't tell the difference between your new feature introductions and your maintenance fixes, and that should trigger some self-reflection. He clearly couldn't let all of the thousands of different kernel components operate in the style you seem to prefer for bacachefs.
If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.
Concerns about general policy need to be handled separately from individual cases. You needed to play along, advocate for your position, explain your position (e.g. demonstrate how your processes eliminate certain classes of errors; or find some objective way to measure QA problems and present the stats), and push for a holistic process change that supports your use-case. That process change would need input from other people, and might end up very different to your original proposal, but it could have happened. Instead, you've burned (and continue to burn) bridges.
Linus's job is not to make the very next version of the kernel as good as it can be. It's to keep the whole system of Linux kernel maintenance going. (Maintaining the quality of the next version is almost a side-effect.) Asking him to make frequent exceptions to process is the equivalent of going "this filesystem is making poor decisions: let's hex-edit /dev/sda to allocate the blocks better". Your priority is making the next version of bcachefs as good as it can be, and you're confident that merging your patchsets won't break the kernel, but that's entirely irrelevant.
> If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.
> Concerns about general policy need to be handled separately from individual cases.
Citation needed.
> Linus's job is not to make the very next version of the kernel as good as it can be. It's to keep the whole system of Linux kernel maintenance going. (Maintaining the quality of the next version is almost a side-effect.) Asking him to make frequent exceptions to process is the equivalent of going "this filesystem is making poor decisions
You're arguing from a false premise here. No exceptions were needed or required, bcachefs was being singled out because he and the other maintainers involved had no real interest in the end goal of getting a stable, reliable, trustworthy modern filesystem.
The discussions, public and private - just like you're doing here - always managed to veer away from engineering concerns; people were more concerned with politics, "playing the game", and - I'm not joking here - undermining me as maintainer; demanding for someone else to take over.
Basic stuff like QA procedure and how we prioritize patches never entered into it, even as I repeatedly laid that stuff out.
> > If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.
> You have missed the point by a mile.
No, that is very much the point here. bcachefs has always had one of the better track records at avoiding regressions and quickly handling them when they do get through, and was being singled out as if something was going horribly awry anyways. That needs an explanation, but one was never given.
Look, from the way you've been arguing things - have you been getting your background from youtube commentators? You have a pretty one sided take, and you're pushing that point of view really hard when talking to the person who's actually been in the middle of it for the past two years.
Bureaucracies run on process. You call that "politics", but (when arguing with programmers) I call it code. The Linux kernel project is a bureaucracy.
> citation needed
I'm sure you're familiar with "separation of concerns" in programming: it's the same principle. My experience of bureaucracies is experience, but I'm sure most good books on the topic will have a paragraph or chapter on this.
> bcachefs was being singled out because he and the other maintainers involved had no real interest in the end goal of getting a stable, reliable, trustworthy modern filesystem.
I imagine they would dispute that.
> No exceptions were needed or required,
I know that Linus Torvalds would dispute that. In this HN thread and elsewhere, you've made good arguments that treating your approach as an exception is not warranted, and that your approach is better, but you surely aren't claiming that your approach is the usual approach for kernel development?
> undermining me as maintainer
Part of the job is dealing with Linus Torvalds. You're not good at that. It would make sense for you to focus on architecture, programming, making bcachefs great, and to let someone else deal with submitting patches to the kernel, or arguing those technical arguments where you're right, but aren't getting listened to.
People are "more concerned with politics" than with engineering concerns because the problem is not with your engineering.
> bcachefs has always had one of the better track records at avoiding regressions and quickly handling them when they do get through
That's not relevant. I know that you don't see that it's not relevant: that's why I'm saying it's not relevant.
> have you been getting your background from youtube commentators?
No, but I'm used to disputes of this nature, and I'm used to dealing with unreasonable people. You believe that others are being unreasonable, but you're not following an effective strategy for dealing with unreasonable people. I am attempting to give advice, because I want bcachefs in the kernel, and I haven't a hope of changing Linus Torvalds' mind, but I have half a hope of changing yours.
Rule one of dealing with unreasonable people is to pick your battles. How many times have I said "dispute" or "argument" in this comment? How many times have these been worthy disputes? Even if I've completely mischaracterised the overall situation, surely you can't claim that every argument you've had on the LKML or in LWN comments has been worth bcachefs being removed from the kernel.
Look, it wasn't just this one thing. There had been entirely too many arguments over bugfixes, with the sole justification on Linus's end being "it's experimental garbage that no one should be using".
I can't ship and support a filesystem under those circumstances.
The "support" angle is one you and a lot of other people are missing. Supporting it is critical to stabilizing, and we can't close the loop with users if we can't ship bugfixes.
Given the past history with btrfs, this is of primary concern.
You've been looking for a compromise position, and that's understandable, but sometimes the only reasonable compromise is "less stupidity, or we go our separate ways".
The breaking point was, in the private maintainer thread, a page and a half rant from Linus on how he doesn't trust my judgement, and then immediately after that another rant on how - supposedly - everyone in the kernel community hates me and wants me gone.
Not joking. You can go over everything I've ever said on LKML, including the CoC incident, and nothing rises to that level. I really just want to be done with that sort of toxicity.
I know you and a lot of other people wanted bcachefs to be upstream, and it's not an ideal situation, but there are limits to what I'm willing to put up with.
Now, we just need to be looking forward. The DKMS transition has been going smoothly, more people are getting involved, everyone's fully committed to making this work and I still have my funding.
It's going to be ok, we don't need to have everything riding on making it work with the kernel community.
And now, you guys don't have to worry about me burning out on the kernel community and losing my last vestiges of sanity and going off to live in the mountains and herd goats :)
This is exactly how technical debt accumulates. You can write a comment documenting that "unsafe is actually safe here in routine X because Y will always do Z," but if the code lives long enough, someone may eventually change X or Y in a way that falsifies that claim, or try to cut-and-paste X's code into a new context without those guarantees. No, that's not /prudent/, but it happens nevertheless, and the costs of tracking down and fixing the errors are almost always higher than just implementing more conservatively in the first place.
I've often wondered how the evolution of C and C++ might have been different if a more capable preprocessor (in particular, with more flexible recursive expansion and a better grammar for pattern matching) had caught on. The C++ template engine can be used to work around some of those limits, but always awkwardly, not least due to the way you need Knuth's arrow notation to express the growth in compiler error message volume with template complexity. By the time C++ came out we already had tools like m4 and awk with far more capability than cpp. It's pretty ridiculous that everything else about computing has radically changed since 1970 except the preprocessor and its memory-driven constraints.
Amdahl's Law, like most 20th-century performance "wisdom" and metrics, focuses excessively on instruction count, neglecting memory and I/O pressure. 64 cores doesn't mean 64 independent cache pyramids and memory buses. In real life, the difference between CPU cycle frequency and memory latency is so great that memory pressure primarily determines performance, whereas core count really only matters to the extent that it contributes to that memory pressure.
Right, because every executive who pursued offshore manufacturing was thinking, "gosh, how can I deliver even lower prices and better value to my customers?", and not "OK, we've shown the market will pay $X for product Y, how can I cut my costs and free up more money for bonuses and cocaine?"
Graphs of price indices (aside from a few sectors such as electronics where it was the core technology that improved, not labor efficiency) and wages over the last 50 years clearly show that the bulk of any offshoring savings were not passed along to consumers or front-line workers.