The real problem is the Redis Cluster protocol, which pushes all the smarts onto the client. This means each client implementation behaves differently and fails differently.
I would encourage Redis users to use Envoy Proxy as the Redis client (ie use vanilla client, and use Envoy as the cluster client). You get all the HA and usefulness of Redis Cluster, but way less of the headache. Also, strongly encourage people to check out Elasticache, which is really good.
And to your point about client libraries: for any shared behavior that needs to be local to the app (eg. session handling, permissions, etc.), it's more appropriate to spin up a sidecar that you can talk to over sockets than to try to build client libraries in each and every language. Your client libraries will differ in behavior and it is an incredible pain keeping them all up to date, patching every app, etc.
Envoy for S2S and traffic mesh + sidecars for shared behavior is better than building client smarts.
But the Envoy proxy only supports a subset of the commands. For example, it doesn’t support LPOPRPUSH, precisely because the keys could be stored in multiple nodes, and they don’t support that part of the clustering protocols.
We (me and some folks at my old consultancy) wrote an Erlang version of Redis (https://github.com/cbd/edis) for some of the same reasons - multithreading changes some of the scaling semantics in interesting ways. It was mostly for fun but ended up in some real projects as a simple REDIS protocol implementation front-end where the backend could be replaced with whatever the implementor wants.
Does anyone have any experience with these other Redis clones? I need to write a benchmark on these someday (the outline for the blog post is already written), but have restricted my yak shaving recently:
Yes. KeyDB is legit, it works well, it's free speed in a lot of cases, the maintainer is a nice guy and is very responsive. We used it in production and it had the same robustness you'd expect from Redis.
I recommend keydb in almost every case redis is used.
Yea FLASH is only in the Enterprise version. At a rough level the open source is the best cache we can make while pro/enterprise is the best full featured database. It’s not fully there yet but the end goal is you can get caching and persistence in one place.
The FLASH feature in Enterprise can send all writes to disk before replying +OK, however notably we don’t fsync (AOF is still recommended if you need that). In these deployments it’s not necessary to use RDB or AOF except for replication.
And of course we have all the existing options already in Redis.
Someone on my team looked at it, and for our workload (lots of MGETs per query) it wasn't that much better than redis. We may still go with it though, our GCP Memorystore instance is pretty much at 100% CPU constantly.
Long running commands are still a problem. We have infrastructure in the Enterprise version to alleviate this but so far we’ve enabled it only for KEYS and SCAN. MGET does come up frequently so it’s on our radar.
It's multithreaded but what about Redis running on a single core in a cluster? Like running 8 Redis on a single 8 cores CPU. I don't really understand the reason to run Redis on multiple core since you can run multiple Redis on a single CPU with clustering which will have better perforamce than running KeYDB with the same number of cores.
Take the scenario where you have an 9 node redis cluster. With KeyDB you could shrink that down to 3 nodes with 3 threads while getting the same throughput. This reduces the maintenance burden while also reducing latency. There's overhead to connecting to a different cluster server each time you access a seperate shard.
Even better you might be able to avoid having a cluster at all. For many that’s the biggest win with KeyDB.
This what redis’s creator antirez suggests: “ the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster” http://antirez.com/news/126
It’s basically back to the multi-threading vs multiprocessing debate that also exists in Python. With multiprocessing, you have more overhead but the client itself is simpler.
It’s kind of a weird in between where you need more scalability then a single redis but no so much that you want to go to a cluster.
Are you concerned about AWS starting a competitor to keydb cloud/have you considered modifying your license to prevent that from happening? I'd imagine that'd be important in ensuring the long term sustainability of keydb development
They have Elasticache which is always a concern. In terms of open source projects my read is they really don’t want to run their own and are much more comfortable operating open source as a service. They “forked” Elastic Search 2 years ago and basically did nothing with it until it was re-licensed a few months ago. Now that they are investing more into it I’m interested to see how they handle it long term.
If you switch to the impact view, you can see it's pretty much one guy doing all the work right now. The impact view also shows 1 frequent, 1 occasional and 13 seldom contributors, so I'm guessing the number of people working on OpenSearch is quite small.
Note, it is also quite possible that a lot of the work is being done behind the scene, so looking at the OpenSearch repo may not tell you the whole story. And if you search for OpenSearch in amazon's job board, you find they are hiring so I guess we'll have to wait.
Note: Do not install GitSense as the docker image has an out of date license that I'll need to update when I get the time.
This seems somewhat orthogonal to the question asked. They have Kinesis which competes with Kafka, SQS with competes with a variety of message queues, redshift, dynamo db, etc and it’s my understanding that Athena was originally a fork of Presto. So, they could definitely move more heavily into the caching space if they see there’s demand.
Elasticache is already that competitor. They are scary to be sure but they have some weaknesses. The main one is branding: Elasticache is a white label Redis not it’s own thing, to the point where their own docs use the two interchangeably.
I can't find in documents but does multi-threading effect consistency somehow? Is there a chance that I wouldn't read what I just wrote? I'm talking about single node, not about replication, cluster etc.
If it provides same consistency, is threading like :
Yea that was roughly KeyDB’s original design. There’s some nuance in the locking like ensuring it’s both fair and fast so P99 latency doesn’t get out of control.
In the Enterprise codebase we can take snapshots which lets us do reads without the lock but it’s a bit of work to enable for commands so it only applies to KEYS and SCAN at the moment.
KeyDB uses a custom lock I wrote called the “fastlock”. At its core it’s a ticket lock with some tweaks to better tune it to KeyDB. When uncontested it’s a single atomic increment. When contested ownership changes extremely fast. If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find.
When I first tried making KeyDB I used posix locks. Those were way too slow.
Generally speaking you don't want to be dealing with this stuff unless you really have to. For 99.999% of people the locking primitives that come with your programming language are good enough.
"My thought process was simply that there is a big need here and Redis had for some reason decided not to serve it. If they won’t then we will."
I feel like this and the general tone of the article are needlessly antagonistic toward Redis. KeyDB is building their entire business off of it after all.
There may very well be a need for multi-threaded Redis, but Redis as it stands today is an amazing project and there's something to keeping it simple along the lines of the project philosophy.
I’ve learned to appreciate Salvatore’s stance on simplicity the longer we’ve gone on with KeyDB. But when I first made KeyDB two years ago I was really perplexed at the decisions he made with respect to threading.
It’s not my intention to be antagonistic. I’ve had a lot of projects over the years that went nowhere and a part of me is sad that the one with the most traction is a fork.
I am aware. Community management is different from what is legally possible. Were a project owner start to act in a self-interested or malicious manner, then I think proudly and aggressively forking a project is a great idea. That's not Redis. Like I said, it may be a good idea to have a multi-threaded Redis, but Redis users tend to love Redis. I would probably lean into that goodwill instead of against it.
I don't see forking as being aggressive. It's the natural thing to do if you want to take the project in a different direction to its stewards. Often lessons are learned from that process and the learnings integrated back into the main project.
I'm curious about the name. Putting "DB" in the name sort of suggests it might support persistence, more data than fits in memory, write-through, etc. Is that the case? Or is the "DB" some nod that clustering means that "in-memory" doesn't have to be ephemeral?
We have our FLASH feature which gives us a baseline persistence story, you don't need RDBs or AOFs anymore. However there's a lot of work left to do in this area.
The long term goal of KeyDB is to let you balance your dataset across memory and disk in one database. In the future I think caches will just be a feature of a more full featured database and that's where we're heading with KeyDB.
I don't think more data than fits in memory is a requirement. An important innovation in databases has been the realization that there are important performance benefits from assuming all data can fit in memory even if you still provide persistence.
Even persistence isn't really required. A pure analytics view of another database is still a database by itself, but it doesn't need to actually persist anything. It seems like querying is more important to the concept of a database rather than actual storage.
Redis is one of those things where you see a car with an after-market door on it, but the door is actually a house door, cut with a sawzall into the shape of a car door.
Would this mean that in a single core or low end environment, it would be better to use Redis. I'm assume cutting out multi-core complexities would be beneficial.
This is totally going to be a Hacker News Bingo type of question.
But has anyone tried to do a clean room implementation of Redis using Rust, but speaks the same wire protocol? You would get the zero-cost multi-threading, memory safety, etc, and it would be a drop in replacement.
I think you mean zero cost abstractions. Which aren't usually zero cost, but just zero additional cost over doing it yourself.
There's no such thing as zero cost multi threading. Just tradeoffs. Rust actually doesn't help with performance here (it gets in the way often) but it definitely does help with correctness - which is truly hard with multi threaded programs.
I mis-worded. I meant safe multi-threading using rust’s abstractions that make this a lot easier and guaranteed, and zero cost meaning no overhead. Is that not the case? maybe not the case with threading.
I've done a C clean room version and I will say that the networking part is as important as the multi-threading the data structures:
https://github.com/raitechnology/raids/.
If you go to the landing page of the above, scroll down to the bottom, there is a TCP bypass solution graphed, using Solarflare Open Onload and it is capable of running several times as fast as the Linux Kernel TCP. I didn't test Redis with Open Onload, but I'm pretty sure you'll get a similar results since TCP is a major performance bottleneck in Redis as well.
No, I don't believe that's what it means in this case. I took the usage to mean that since Rust has memory model which is different enough from C to require a redesign.
I looked at many implementations of Redis and read many KV papers. My redesign reason was similar to a "clean room Rust" reason, I desired a memory model that used shared memory that was independent of the protocol (Redis RESP in this case), allowing multiple processes with different types of protocol services to attach to it.
It's been a long time since I've looked at KeyDB, but IIRC KeyDB is just Redis plus a spinlock. It's actually still very performant. There are other "toy" reimplementations of Redis in Rust that take the same approach and aren't even as performant as single threaded Redis.
The next approach you could take is using something like Glommio and take a thread-per-core design to Redis. I think that approach has a lot of potential, but the design becomes more complex (you now need something like distributed transactions for "cross-core" Redis transactions and mutli-gets)
"ScyllaDB also handles complex query processing in the same engine. RonDB does all complex query processing in MySQL in a separate program." — i.e., as soon as things might get a bit hairy, RonDB punts it off-system. While a clever way to keep performance numbers high, you'd have to also then analyze the MySQL latencies and throughputs to get an overall view of complete system performance.
It's a clever dodge, but it's not a magical way to eliminate the need for those pesky CPU cycle times.
I would encourage Redis users to use Envoy Proxy as the Redis client (ie use vanilla client, and use Envoy as the cluster client). You get all the HA and usefulness of Redis Cluster, but way less of the headache. Also, strongly encourage people to check out Elasticache, which is really good.