“just noting” that this has no relevance to a sober and objective review of the points made within. i sometimes think that we’re lucky we still have two sides of the aisle in this country. i constantly think we ought to aspire to more
Any old timers here in the thread, please email me at strep_makers.0p@icloud.com — I’m building a new company doing both consulting and internal product development and have always loved working with older, more experienced folks. We’re looking for tech leads, technical project managers, and anyone who has the ability to take strong ownership.
(FYI, that’s a burner email— I’ll reply from a different one)
I made Redwood, a general purpose p2p/distributed database (and blob store) with pluggable merge algorithms (including some cool CRDTs, like what @josephg is working on).
Oh, hey there, Coventry :)
I’d be curious to know the same, as well as to hear a comparison between Algorand and:
1) recent work on aBFT protocols, many of which don’t seem to require any kind of staking, and...
2) Chainlink’s off-chain aggregation protocol’s consensys mechanism
Redwood is a decentralized database providing a data model that's much like a Redux state tree. One of the fundamental design goals was to better harmonize how data is handled on frontend and backend layers.
It's also:
- syncable/offline-capable, thanks to a CRDT system
- able to serve as a standalone Git backend
- embeddable into Go applications
- immutable, in a blockchain-y sense, due to its append-only ledger
There are some demos in the Github repo:
- A real-time collaborative text editor
- A chat application
- Git integration
- A video streaming application (still improving performance here, but it's functional)
It's pretty alpha still, but feedback and contributions from anyone interested would be very welcome.
A few projects I'm involved with that might be worth sharing here:
- Redwood (https://github.com/brynbellomy/redwood), a realtime, p2p database. Data is structured in state trees that evolve over time. Merge algorithms are configurable at any point in a given state tree, but the default is a CRDT algorithm called "sync9".
- Braid (https://github.com/braid-work and https://braid.news), a draft IETF spec that specifies certain extensions to HTTP that make it much easier to build systems like Redwood. The Braid spec is under active development on Github, and we welcome input from anyone interested in the idea.
- Axon (http://axon.science and https://github.com/AxonNetwork), marketed as a platform for making it easier to collaborate on scientific research, but under the hood, it's basically just some extensions to git that allow you push commits over a peer-to-peer network bound together by a DHT.
I would also highly recommend getting involved in the Internet Archive's DWeb meetups and conferences, where you'll find hundreds of people interested in solving exactly these kinds of problems.
Creator of Redwood here, which is similar in many ways to this and GunDB (hi Mark!). Agreed that we need more projects like this.
Redwood might be better described as a "p2p application platform" than a realtime database, but there's plenty of overlap. It's written in Go, supports HTTPS, Libp2p, and WebRTC transports (and soon, a Cap'n Proto RPC interface). It can also:
- serve as a Git remote
- stand in as an application server (you can upload assets directly to it, and they're served over regular HTTPS)
- execute Lambda-like functions written in Go, Lua, and Javascript that can be installed at any point in a given state tree
Would love some early feedback, and also to discuss similarities and differences with Sapphire's model. I've set up some demos in the "scenarios" folder -- a chat app, a realtime document editor, and a Git integration. Take a look if you're interested:
This is interesting, and good to hear. Are there still a lot of projects adding capnproto support? It's a great protocol but I've worried it might be dying.
Dying? Hm, not to my knowledge. In fact, I saw a highly-commented thread on HN just a couple of weeks ago about Cap'n Proto. What gives you the sense that it's on its way out?
Sandstorm Oasis going away and the last time I checked (maybe a year ago?) work seemed to have stopped on getting the protocol implemented in the browser.
Hi, I'm the author of Cap'n Proto, and co-founder of Sandstorm.io. When Sandstorm failed, I went to Cloudflare where I'm leading the Cloudflare Workers project. We're using Cap'n Proto (including RPC) pretty heavily there!
That said, it's true that Cap'n Proto relies on its users to contribute improvements to the implementation. We don't currently have a team who can dedicate their time to building out capnp implementations in every language.
At present, within Cloudflare, the heaviest use is C++, with some stuff in Go, Rust, Java, and Lua. Hence, if you look at the C++ implementation you'll see it's been pretty active. At the moment we don't have a JavaScript use case so haven't put effort into that implementation. But given that Workers itself is a JavaScript platform, it seems reasonably likely that we'll end up putting some work into the JavaScript/TypeScript implementation at some point down the road.
Glad to hear it! It really would be a shame for such a great protocol to go unused.
That's interesting that you feel Sandstorm failed. I draw inspiration from it on a regular basis for several of my projects. Maybe it was just ahead of its time.
Sandstorm the company definitely failed (ran out of money and couldn't raise more, while having negligible actual revenue). Sandstorm the open source project continues and has actually seen an uptick in development activity lately.
Nice project! I had considered building something similar (actually just services discoverable via DHT, nothing concerning storage). Can you discuss here or in the README its resiliency toward bad actors? Specifically, how does it work in a p2p environment with untrusted clients? If it's still an unknown, just a note on whether it's even a threat model you're developing within would be helpful.
Sure! Nailing down a robust security model is a major goal for Redwood. There are currently a few different mechanisms, some already partially built, and others still in the research stage.
For what it's worth, I have a silly side project goal of reimplementing Bitcoin on top of Redwood, just to prove it can provide the same (general) security guarantees. Not sure I'll get around to that any time soon, though :)
Anyway, here's what we do:
1. Users are identified by a public/private keypair (actually using Ethereum's implementation at the moment, just so I could focus on other stuff). All transactions must be signed by the sender, allowing recipients to verify the sender identities.
2. You can place "transaction validators" at any node in your state trees, and any transaction affecting the subtree under the validator will be checked by that validator. Currently, there's a "permissions" validator included that gives a simple way to control writes based on the Ethereum keypair I mentioned above. The current implementation of the permissions validator isn't very well fleshed out yet -- it just lets you specify a true/false write permission associated with a [user, keypath regex] pair -- but it's a decent placeholder for now, until I can get around to roles and policies.
3. You can also write custom transaction validators in Go/Lua/Javascript, which should make it trivial to implement just about any access control model you desire.
4. You can also create a "private" tree by explicitly specifying the set of users who are allowed to read from and write to that tree. The default Redwood node implementation does automatic peer discovery and keeps a list of peers whose identities/addresses have been verified. When it receives a transaction for a private tree, it only gossips that transaction to the tree's members (as opposed to its behavior with public trees, which is to gossip transactions to any peer who subscribes to that tree).
5. We also want secure persistent storage for transactions so that we can get high availability and redundancy without compromising the security model. To facilitate this, we allow you to configure the node to talk to what we're calling a "remote blind store" (essentially a key-value DB running on a separate piece of hardware, possibly off-site). The Redwood node encrypts transactions with a key the remote store doesn't possess and sends it over gRPC to the blind store.
6. Some applications will have a high volume of transaction data, and will want to be able to prune/merge/rebase that data to save space. To prevent bad actors from issuing malicious prune requests, the remote blind stores are going to implement their own p2p protocol involving threshold signatures. Prune requests will only be honored if a quorum of these blind stores sign off on them.
We've been thinking about this quite a bit, but not publicly. I'd love for someone to help us punch holes in it!
I've been working for a few months on a database called Redwood that's intended to make it easier to build this kind of software. Having spent much of the past couple of years working with libp2p, IPFS, Dat, and similar technologies, I was curious to see what would result if I started from the ground up.
So far, the model seems promising. It's fully peer-to-peer, and supports decentralized identity, configurable conflict resolution, read/write access, asset storage, and currently is running across 3 different transports:
- HTTP (augmented with extensions proposed by the Braid project [1][2])
- Libp2p
- WebRTC
I've included two simple demos, a collaborative document editor (well, it's just a textarea at the moment), and a chat app. Would appreciate any feedback or participation that folks are willing to give.
We’re building something along these lines at Axon (http://axon.science). Sign up for our beta if you’re interested in checking it out, and we should be able to get you set up in the next few days (we’re just starting to roll things out to the public this week).
The basic idea is distributed version control, like git, but over p2p swarms rather than clusters around “central” repositories. We have special handling for large datasets (but still using git) to improve transfer efficiency and diffing.
There’s a UI layer for collaboration (discussion, PRs, review) that supports deep linking to and embedding of files at specific commits, which sounds a bit like what you’re looking for.
That looks very interesting, particularly the UI layer for collaboration. Your website says it supports “massive data sets” but I would spell out what you mean since data for different fields vary by several orders of magnitude. (Massive for me starts at TBs and goes to petabytes.)
One of the issues for me is file-based versioning, which then requires the means to parse the format. A number of ventures and organizations (e.g., NeuroData without Borders) address versioning of the entire ecosystem necessary to correctly use the underlying data files, so not sure if that’s an explicit part of your ecosystem. Most importantly, is your stack going to be open source?
