Cool stuff as usual, Anders. One of the nice things about running a networked DB is that it makes redeploying the application a bit simpler. You can spin up a new EC2 instance or whatever and once it's online kill the old one. That gets 0 or close to 0 downtime. If the DB is on the same instance, replacing it requires loading up the DB onto the new instance, which seems more error prone than just restarting the app on the original instance, but in my experience that typically incurs downtime or some complicated handoff logic. Have you had to deal with anything like that running sqlite in prd?
You're going to want persistent storage on your server, not ephemeral. You'll also want NVME. A lot of the time you're going to end up on bare metal running a single server anyway.
You're going to have down time for migrations unless you're very clever with your schema and/or replicas.
Litestream for me at least is what makes SQLite viable for a web app as prior to that there wasn't a good replication story.
With litestream it's much easier to have a backup on standby. That being said where I have used it in production some amount of downtime has been acceptable so mileage may vary.
> You're going to have down time for migrations unless you're very clever with your schema and/or replicas.
probably worth stating these kinds of design considerations/assumptions up-front
i'm sure lots of applications are fine with "downtime for [database] migrations" but lots more are definitely not, especially those interested in synthetic metrics like TPS
I'd argue the opposite most applications are fine with an hour of downtime a month and arguably much more downtime then that. The recent AWS and Cloudflare outages have proven that.
You can achieve zero downtime with Sqlite if you really need to.
TPS is not a synthetic metric when you cap out at 100 TPS because of Amdahl's law and your users having a power distribution.
1h of downtime per month means you're delivering at best two 9s of availability. again that may be fine for lots of applications but it's trivial scale, and certainly a couple orders of magnitude below what aws and cloudflare provide
taking a step back, if your application's db requirements can be satisfied by sqlite [+replication] then that's great, but that set of requirements is much narrower, and much easier to solve, than what postgres is for
Personally, I'm a fan of event sourcing if you need zero downtime you probably need auditibility. You have one SQLite database that's an append only event log. You build projections from it into any number of SQLite databases. These databases pull from the log database and update themselves. They are completely expendable. So you never have to vacuum them or migrate them. You just build a new projection and then point to it. This is also how you can spread your sqlite over nodes (if that's your thing, with something like NATS).
There are other ways where you're just more disciplined with your schema and indexes. I.e jsonb, partial indexes and existence based programming (from data oriented design).
That's indeed a possible (and very resilient) solution. But that's a significant shift for many apps. I'm fan of event sourcing in theory, but I've always been afraid of it making things overly complex in practice, for relatively small apps. But I haven't tried hard enough to have a real opinion on this.
Oh for sure. That's why my default is if you don't need event sourcing and are ok with some down time a month if you need to add a really large index. Keep it simple.
Once you have a decent CQRS set up (which is a natural fit with SQLite) and event sourcing it can be quite easy to spin up on a new project.
I think people don't have an honest assesment of what their project requirements are in terms of uptime and scale. So they start with crazy microservice multinode architectures that are designed to scale to the moon and inevitably end up with more downtime due to complexity.
I'd still recommend people start with managed PG for most things. But, if you're comfortable using VPSs or bare metal servers SQLite can take you very far.
I fully agree that most projects would work perfectly fine with a monolith and PostgreSQL. And yet they go with microservices, Redis, RabbitMQ, etc, making the system more complex and less available.
SQLite would be perfect if it would allow multiple concurrent writers, which would allow running database migrations without downtime. I’m looking forward to Turso for this.
Sqlite supports multi-process access, so as long as you've configured to permit that, you can do the same “start new process before retiring the old one” dance.
I recently used c.a.flow for a program that reads files from s3, massages the data a bit, and writes to a database. The code came out very easy to understand. The challenges were:
- getting back pressure right. You need to tune the buffers for the input chans correctly, which takes some thinking, and in my case, some blowing up in prd.
- flow monitor (the UI for debugging flows) can't handle large process states, which makes it pretty much useless for my program.
- understanding the flow as a whole (rather than specific processes, which are easy to understand in isolation). For example, answering questions like "why hasn't this process received a message for a while?" was tricky.
Thanks for the feedback! Flow monitor does now support filters on the process state (and more on that it is coming to flow itself soon). If you were able to use monitor, it shows the channel buffer states, I guess that was not sufficient to guess why values weren't flowing?
It's always frustrating to me that the order matters at all. I understand SQL can be complex to parse, but surely we are smart enough to come up with a parser that doesn't care whether FROM is before or after SELECT?
HoneySQL lets us define queries with maps, like {:select [:col1 :col2] :from :table}, and turns that into SQL. In a better world, SQL would be structured data like HoneySQL, and the strange SQL syntax we know and love would be a layer on top of that, or wouldn't exist.
This post could be summarized as "write as much of your project's tooling as you can in the project's main programming language and the project's main programming language should be Clojure" and I agree wholeheartedly.
I think their point is bigger than that. Historically one of the major points against using Clojure to write your tooling was the slow startup times which are just painful from the CLI. It looks like the clj-exec idea linked to in the article is the secret sauce that makes moving to writing your tooling in Clojure a workable idea, since now you have a unified calling convention for both calling tooling from the REPL during development but also activating your tooling from the CLI in some CI or build pipeline where the massive Clojure startup times don't matter.
This article comes at a perfect time, we're just starting a new Clojure project and were looking into how to automate tooling since we'd been burned by Clojure startup times before. Looks like clj-exec means we can now unify our work on Clojure.
Also check out https://babashka.org: it offers Clojure scripting with very fast startup time. It also has a task runner (similar to make, just, etc.) that can be used to store long invocations (like clj-exec tends to have).
Long ago, when Datomic was only recently released, I saw blog posts (not from Cognitect) claiming that Datomic's transaction log could be used to create a subscription system for arbitrary datalog queries. Basically "tell me when the result of this query changes". In practice, I've yet to find a general way to do that, except for wildly inefficient stuff like keeping query results in memory, requerying on each transaction, and comparing. How does 3DF handle a query like this?:
[:find (max ?num) :where [_ :attr ?num]]
It seems like Datomic's transaction log can only tell us that the result of this query might have changed. Does differential dataflow solve that problem?
I think you would need to use only datomic's transactor and tx log and then build your own incrementally maintained query engine, which is basically what this project declarative dataflows does. Differential dataflow and regular dataflow are both ways to implement this. Differential dataflow has the restriction that your query engine must be implemented in terms of diff-aware primitives (https://news.ycombinator.com/item?id=22095199). Regular dataflow is like Excel; it works with any functions, but is not as good at incrementally maintaining things like large lists and nesting – if you're familiar with React.js, at a high level, regular dataflow works a lot like React.js rendering which can call any function during rendering. (Note React.js is highly specialized to html view maintenance, not general purpose incremental maintenance of any computation). Someone correct me if I've said something wrong.
I've been guilty of this. Normally I like to blame CSS for all of my personal failures, but I think the definition of "overflow" is intuitive enough. I was hesitant to turn on visible scrollbars system-wide because I thought it might be ugly. I did and it's not, so I second the author's suggestion to do that.
This seems like an excessively uncharitable read of the situation. I've never used Nubank's software, but I have used (on-prem) Datomic and I certainly wouldn't say it has bugs everywhere. In fact, in my (admittedly low-volume and simple) usage of the system I haven't come across any bugs I can remember. Calling Cognitect a "burnout" company is inaccurate and rude.
I agree with you that the Datomic cloud stuff comes across as being frighteningly complex. I think they probably just need to work on the documentation, like making it more obvious what the differences and tradeoffs are between the deployment scenarios.
Did you inherit a Datomic system that was previously developed by a small team or a small company? Because inheriting a system that's hard to understand and change transcends languages and databases. It is the tie that binds us all as software developers.
Fair enough, hitting that bug would have pissed me off too.
On your last point, I agree that it still has a way to go. It's good for some (many?) production use cases now, as Nubank's success demonstrates, and hopefully with Nubank's resources it'll start to live up more to its promise.
It sounds like Nubank might have the biggest Datomic installation in the world (2000 servers). This is exciting news for people like me who think Datomic is the most interesting database out there. Putting more resources behind Datomic might finally allow it to go mainstream.
I was unable to bring Datomic in-house. Mainly because our organization has a huge agreement with Oracle and all our database servers are essentially "free" to teams.
So I cheated. I created some schema designs that were immutable. I added a GUID, timestamp, and a deleted flag (value of 'Y' or 'N') to tables. Basically, all selects are against views that are defined over the tables to select the tuple associated with the max(timestamp) for that tuple along with the tuple having a deleted flag value of 'N'. This means that any select only sees the most recent tuple value for a GUID if it hasn't been "deleted".
There was a little bit more hiding in there to handle dirty writes.
But this worked very well for my requirements. By really only doing inserts, I was able to do similar "point in time" looks at the db as an immutable value.
Would you be willing to have an email chat about that? I'm thinking about doing a similar strategy with our relational database and wanted to get your experiences with that.
