This looks neat. I'm the author of a similar project in typescript we use at Cotera called Era [0]. Y'all might be implement something similar to our caching layer [1] which we think is super useful. Once you have a decent cross warehouse representation it's pretty easy to "split" queries across the real warehouse and something like duckdb. The other thing that we find useful in Era that y'all might like are "Invariants"[2]. Invariants work by compiling lazily evaluated invalid casts into the query that only trigger under failing conditions. We use "invariants" to fail a query at _runtime_, which eliminates TOUTOC problems that come from a DBT tests style solution.
In my mind I see the problem of dynamic linking in rust to have a bunch of overlap with the "I want this rust library to be exposed in my higher level GC'd language with minimal safety issues/tedious handmaintained bindings" problem.
My hunch is that the lack of expressiveness of the C ABI is holding back both. the thing I'd love to see some sort of "higher level than the C ABI" come out. And something like `wasm-bindgen`[0] to exist for more languages.
Here's a link to the rust "interopable_api" proposal! I don't understand all the implications, but it seems to be in the right direction https://github.com/rust-lang/rust/pull/105586
The result of Microsoft developers in the 90s drunkenly making up an object model on top of C with the goal of it being cross-language and cross-platform (in theory). The idea being that you can define an interface and implement it in any language and consume it in any language. A good idea in theory. In practice, DLL hell often made the experience unpleasant. A big part of Windows's APIs are COM-based. For historical reasons it's also used at the core of a few macOS system components, hence the grandparent comment
COM is the main way Windows APIs are exposed since Windows Vista, as the .NET ideas for Longhorn were rebooted as COM.
While plain old bare bones Win32 APIs are still coming up in every release, they aren't the main ones.
Also COM as concept wasn't something designed by Microsoft, it rather build up on the ideas of what Sun, IBM and co were doing with distributed objects protocols.
It is no coincidence that COM IDL language is based on DCE RPC.
At the most basic level (IUnknown), COM doesn't care about DLL loading at all, so DLL hell is not in the picture. Many Windows libraries are actually like that - you call a single global exported function that gives you an "entrypoint" COM object, and from there you call methods on that and/or on other things that it returns. This is the extent of COM that you would usually see in cross-platform code (e.g. Mozilla's XPCOM, or COM Interop in Mono when running on Linux or Mac).
Beyond that, if we're talking about coclasses etc, one of the things about Win32 COM was kinda sorta solving DLL hell by using UUIDs as primary identifiers for those classes - CLSIDs - and mapping them to actual files on disk via the registry. Thus, you could have things installed wherever, including DLLs with identical names in different folders. A new version of the class would get the new CLSID, so multiple versions could be installed and correctly resolved at runtime.
Of course, that just made it a registry hell instead, where messing up those (global!) entries would break apps that depended on the affected classes. So Windows XP (IIRC) added the ability to register CLSIDs and map them to DLLs directly in the app manifest, allowing for a fully self-contained solution.
Yes, lower-level modern APIs are written in it, but you can write a modern windows GUI app without really needing to touch COM directly because there are other abstractions built on those APIs.
This was not as true of Windows programming 20 years ago.
Not sure how this disagrees with my statement that "you can write a modern windows GUI app without really needing to touch COM directly because there are other abstractions built on those APIs"
COM is several things. Although COM on its own always means Microsoft's version and they invented it, there have been quite a few reinventions over time. Mozilla uses/used their own version called XPCOM inside Firefox for example. It can be thought of as a standardized way to expose objects across languages, compilers, incompatible ABIs, processes and machines. It provides:
1. A standardization of vtables and how to obtain/query them from objects.
2. A standardized memory management protocol based on reference counting.
3. A standardized type description/reflection IDL and binary data format, so dynamic languages can reflect objects to learn what methods/properties/events they support.
4. A standardized protocol for dynamically invoking methods and properties by string and using variant types, for scripting languages again.
5. A whole lot of machinery for auto-generating implementations of dynamic method dispatch for objects written in C++/Pascal/C#/etc.
6. A way to load/dynamically link against objects by UUID instead of library name. In COM the actual location and name of a shared library is (in theory) unimportant.
7. A way to "activate" objects either in-process, which is basically a wrapper around dynamic library loading, or out-of-process, in which case Windows starts an EXE that then exposes objects as a server using RPC.
8. A way to do OOP RPC on objects over the network or a process boundary (DCOM).
9. A protocol for negotiating UI embeddings (OLE) so one component could contribute menus and toolbars that would get merged with the host program's UI.
And a lot more stuff that I've probably forgotten.
It's really a very feature complete and powerful framework for solving common problems you face when writing software made up on components written in different languages and by different teams. For example, an advanced use of COM is to run some objects at one privilege level and others at lower privilege levels. Installers/updaters often do this so the GUI runs as the user, and the engine runs as an administrator.
Most platforms (except the web) have something vaguely COM-like or COM inspired. Apple's stack uses MIG, Mach ports and Objective-C for similar purposes. Linux never really did but the closest equivalent is DBUS. Android has the Binder.
COM doesn't get much use these days outside a few stock use cases because other platforms never implemented it, so modern devs (web+mobile) aren't familiar with it and so we are now seeing a new generation 'rediscover' parts of it in the context of WASM, thinking it's new. However in its heydey COM was ubiquitous. There were large markets of components for sale, distributed in binary form and written in/for many different languages. COM stood in the middle making it all work. It was one reason that Windows was relatively language neutral and friendly to language innovation. In comparison other platforms are quite language and binding unfriendly.
And it wears its biases on its sleeve so no one is likely to be deceived about its positions. No attempt at evenhandedness, in the great philosophical tradition. Much like Russell’s History of Western Philosophy.
~I love when I see programming languages who's first advertised features are implementable in 8 lines of rust~
Edit: ^ the above had the wrong tone. Thanks to dang for pointing it out. What I meant to express was that it's possible to accomplish a similar safety/ergonomics at the library level in rust in not too many SLOC. My personal preference is towards Rust's approach because the type system gives really powerful composable primitives which makes it possible to have the compiler check a wide range of invariants, instead of just the ones that are common/special enough to go into the language itself
(Example edited after comments from mumblemumble)
// The struct is public, but the contents are private, meaning you can't directly access the secret once it's inside the struct
pub struct Secret<T>(T);
impl<T> Secret<T> {
// The only public way to access the secret, returns a new secret
pub fn map<U>(&self, func: impl FnOnce(&T) -> U) -> Secret<U> {
Secret(func(&self.0))
}
}
impl<T: AsRef<[u8]>> PartialEq<&[u8]> for Secret<T> {
// == does the correct thing (and only works for types that would make sense (`AsRef<[u8]>`)
fn eq(&self, other: &&[u8]) -> bool {
constant_time_eq(self.0.as_ref(), other)
}
}
/* Some other file */
use secret::Secret;
// Translated from the example
fn check_mac<T: AsRef<[u8]>>(mac_secret: Secret<T>, message: &[u8], mac: &[u8]) -> bool {
// This returns a new Secret<[u8; 32]>
let computed_mac = mac_secret.map(|secret| hmac_sha_256(secret.as_ref(), message));
// This uses the `constant_time_eq` impl from above
computed_mac == mac
}
Edit: mumblemumble helpfully points out I demonstrated this poorly, so I tried to better demostrate what I was going for in this comment https://news.ycombinator.com/item?id=33764037
This Rust example seems like it misses the point? Rune detects that you're working with data that is marked as secret, and gives you a compiler guarantee that it's defending against some set of known attacks.
The Rust code above depends on the programmer to consistently remember to enforce safety, and to do so correctly every time.
Sure, you could probably implement that as a library. But, "We don't see much value in compiler help with this, a combination of libraries and being careful gets the job done," would be a peculiar position for a rustacean to defend.
Great callout, I haven't had my coffee yet. Here is a version that better shows what I intended
pub struct Secret<T>(T);
impl<T> Secret<T> {
pub fn map<U>(&self, func: impl FnOnce(&T) -> U) -> Secret<U> {
Secret(func(&self.0))
}
}
impl<T: AsRef<[u8]>> PartialEq<&[u8]> for Secret<T> {
fn eq(&self, other: &&[u8]) -> bool {
constant_time_eq(self.0.as_ref(), other)
}
}
/* Some other file */
use secret::Secret;
// Translated from the example
fn check_mac<T: AsRef<[u8]>>(mac_secret: Secret<T>, message: &[u8], mac: &[u8]) -> bool {
// This returns a new Secret<[u8; 32]>
let computed_mac = mac_secret.map(|secret| hmac_sha_256(secret.as_ref(), message));
// This uses the `constant_time_eq` impl from above
computed_mac == mac
}
I think the interesting part of the example is what you _can't_ do in the other file. It's pretty hard to misuse because the return type of `Secret::map` is a new `Secret`, the only way to do `==` on a `Secret<T>` uses a constant time compare.
I guess my main point is that when you have a instead of having to add new things at the language _level_, if I have something as powerful as the rust type system I can implement the same functionality in not much of code.
That covers the one case in the example, but the language goes even further than that in ensuring constant-time processing of secrets, including ensuring speculative execution in the CPU won't expose the data to timing attacks.
I don't know enough about the subject to really evaluate this in detail, but I am more than willing to at least entertain the notion that the problem space is thorny enough that a language-level solution really can do some things that can't be as effectively accomplished with a library solution. Even in a language with a strong compiler like Rust.
It all gets more complicated when you want to pass more than one secret parameter, or the function already returns a Secret - now you need a monad. The key feature seems to be that the code does not need 'map' or anything, the secrecy flag is propagated regardless.
If you use the vim fugitive plugin[1], The `:Gbrowse` command [2] will open your browser to github on the correct file/commit. It also works on visually selected ranges, automatically linking to the range in github
Hey! Thanks for the comment. CodeLink also auto loads lines of code seamlessly in VS Code, JetBrains, GitHub, GitLab, BitBucket etc. It also makes it easy if you don't have the repo locally - it'll check it out for you. If you're not on the right commit/branch it'll swap to it for you. It'll also unfurl the code on Slack (more integrations inbound!) and allow people you share it with to open it up in their IDE or VCS! Hopefully makes life easier for our fellow devs :)
Both of my tech jobs specifically have had non-solicitation clauses to prevent me from doing this type of thing. They both have 1 year lock outs on convincing my friends to quit their jobs
Of course. The question is whether there is some 'creative' way round it. We have enough startups that are creative with the law to weaken the position of individuals, with respect to privacy, employment, etc. Why not to strengthen it?
Wow, the stack in the article feels like a ton of innovation tokens[1].
I hate to be an armchair expert, but I'll do my best to give the _counter_ opinion to "this is a model of a good startup stack".
If you're looking to build a web app for a business on a small team, some guiding values I've found to be successful (that feel counter to the type of values that lead to the stack in the article)
1.) Write as much "Plain Ol'" $LANGUAGE as possible[2]. Where you do have to integrate with the web framework, Understand your seams well enough that it's hard for your app _not_ to work when it receives a well formed ${HTTP/GQL/Carry Pigeon/Whatever} request
2.) Learn the existing "boring" tools for $LANGUAGE, and idioms that made _small_ shops in similar tech stacks successful.
3.) Learn $LANGUAGE's unit test/integration test framework like the back of your hand. Learn how to write code that can be tested, focus on making the _easy_ way to write code in your codebase to be to write tests _then_ implement the functionality
4.) Have a strong aversion from adding new technologies to the stack. Read this [1], then read it again. Always be asking "how can I solve this with my existing tools". Try to have so few dependencies that it would be hard to "mess up" the difference between local and prod (you can go a LONG way with just Node and PostgreSQL).
Some heuristics to tell if you're doing a good job at the above points,
1.) You don't have to prescribe Dev tool choices (shells, editors, graphical apps, git flows, etc)
2.) You can recreate much of your app on any random dev machine, and feel confident it acts similar to production.
3.) Changing lines of code in your code base at random will generate compiler errors/unit test failures/etc
Most every real world software project I've worked on in the SaaS world ended up with "complexity" as the limiting factor towards meeting the business's goals. When cpu/network/disk etc was the culprit, usually the hard part of fixing it was trying to _understand_ what was going on.
Plain may be very successful in their flow, but I'd say most everything in this article runs counter from the ideas that I've seen be successful in the past.
[2] At our shop we'd say "You're a ruby programming, not a rails programmer, your business logic is likely well factored/designed if it could be straight-forwardly reworked into a rails free command line app"
At this point TypeScript is pretty boring tech, and using it on the back-end is an extra compilation step - with the benefits that brings. And it's easy to say use "boring tech", and in some cases it might call for it, but when you need to have applications/services communicating with one another, need high availability, don't want to manage infrastructure, there is an added value that cloud providers give you.
man this comment needs to be top, quite refreshing tbh from the constant cognitive overload we are used to as developers.
instead of innovation token I would call it attention token :) but seems people might mistake it for some crypto coin.
to me i think the reason this is rampant in the SaaS world is that many are burning through millions of VC money every month and they have to justify enormous valuations by inflating problems, creating solutions for problems that used to be solved by boring tech, and then marketing it to developers on social media which creates the illusion that if you are not on the gravy train you are somehow not an expert.
always fascinating to see just how much marketing and politics goes on consciously and subconsciously.
I think the insanity must end, soon enough. It feels like we're moving into a world where insanity and waste is exposed for what it is. Aspirational? Certainly. And I'm sure the pendulum will swing back and forth.
Just ("4: Have a strong aversion from adding new technologies to the stack.") is worth paying you a couple of grand for every start-up to be that reads this and follows the advice. Large numbers of companies absolutely drown in various stack components, sometimes multiple similar ones within the same tiny team.
I agree with all of this, but I've also have come to appreciate bells included packages like spring or django.
I'm currently working on a large Flask App. It really should have been Django from the outset.
There's a different kind of hell in having the almost but not as good version of everything.
I constantly recommend this at work. The specific content isn't super helpful to Saas day to day development, but for me it built an intuition about postgres that has been invaluable. I think once I understood the "heart and soul" of postgres, the heap and the mvcc, many other properties about the database just "clicked" in my head.
The heap is described in 1.3 [1] and MVCC is described in Chapter 5 [2]. I'd also recommend the Postgres docs [3], they aren't quite as accessible as this but are detailed and overall well-written.
I've worked in Erlang/Elixir for the past few years and I haven't had the opportunity to work "in anger"[0] with languages with traditional threads/mutexes/sempahores. This game was a blast and made me appreciate the Erlang's approach to concurrency. The best beginner resouce I've read on Erlang's concurrency model is here https://learnyousomeerlang.com/the-hitchhikers-guide-to-conc...
