That's a great question. It's surprising how few people are aware of the current German data protection laws (where we're based) and the upcoming EU data protection laws aka GDPR.
There are a few ways to address the issue:
1. Don't store any PII on the database, rather only use it to link to data that's stored on-premise in many places. The database has permissioning, and therefore acts as (decentralized) access control logic. Have a TOS with proper legal teeth so that if a database user does store PII on the database, they are liable in the real world.
2. Run an instance of BigchainDB within a region, e.g. within Germany, and comply with the appropriate laws there. Let PII be on the database. But, each node must follow data protection guidelines, similar to how a single centralized entity would, but now do it for each node.
3. Force encryption of all PII, and pray.
(3) is really a non-option. I stated it because many people are saying "just encrypt". But the problem is quantum computing. In 5-15 years quantum computing will be sufficiently easy to access that any encrypted data that's publicly available can be decrypted. You might say "well let's migrate to quantum-tolerant crypto before then" but that doesn't stop a malicious actor from copying encrypted PII now. You might say "let's use quantum tolerant crypto now" but we've seen with most crypto algorithms that it takes years to harden them. Would you trust your PII with untested crypto algorithms? I wouldn't. In short: putting encrypted PII on public nets is a bad idea. Please, please don't do it.
Re (2): this would be a group of people or organizations running nodes together. (If it was just nodes you controlled it misses the point of being decentralized.) You could store sensitive data in this setup, if each person/org had the proper data protection setup. This is not easy, however.
> Also, are suggesting that no sensitive data should ever be stored in a BigChainDB, or I misinterpret #3?
Actually option (2) shows a way to store PII on BigchainDB. But it's not easy. My recommendation is to do (1). And, like my comment before, please please don't do (3) ;)
> I know. I think if I read "scalable" and "blockchain" in the same sentence again I'll scream.
I acknowledge the hype out there.
FYI we were working on blockchain in 2013; long before the hype. We started encountering massive scale problems in 2014 and working on it in 2015, long before the "scale + blockchain" hype. We started work on this. So we're not doing this because it's some fancy combo of buzzwords, we're doing it because we identified a problem years ago, and have been making steady progress to improve it ever since. It was a surprise to me to see BigchainDB to hit HN front page today, since we've been shipping it since Feb 2016!
> Once bitcoin can handle Visa's transaction volume (250M transactions/day) we can talk scale.
We're not trying to improve Bitcoin. We're building our own thing. As we continue to improve the technology, it becomes useful to ever-wider classes of users.
Re CockroachDB: it's a cool technology. The big difference is that it's distributed but not decentralized. That is, the compute resources (in this case mainly storage) are spread across many machines; but the control is in the hands of a single entity / sysadmin. Whereas decentralized means the the control is spread across many sysadmins; and even a few rogue sysadmins won't take down the system.
Oh I don't think it's a hype thing, I just think I don't understand it :D
As I understood blockchains, they are basically linked lists of hashes, yes?
And the decentralization means, that every node hast this list, not just parts of it, so everyone can always check if the list is consistent.
Also, these lists are append only.
The part were every node has a copy AND the list is append only leads me to something that doesn't scale well. It will always get bigger with every action that is appended AND it will always be multiplied by every node in the network.
I'm probably missing something here, but that is my current state of blockchains, haha.
Many people disagree about what blockchain technology is. To me, it' about the characteristics, rather than how they're implemented. I see three characteristics: decentralized, immutable, assets [1].
Under this framing, the "linked list of hashes" is one partial way to achieve immutability. And "every node has this list" is one partial way to get to decentralization is achieved. But that's only part of it. Eg you need to address: what if a node acts badly? And you want a means to create & issue assets.
> every node has a copy AND the list is append only leads me to something that doesn't scale well.
Correct. That's why there is work to scale better, e.g. via sharding by BigchainDB and by others.
"Truth" of course is a red herring. "Claim" is better. And, "digitally signed claim" is better yet.
Then to answer your Q: a good sharding approach should let you see all digitally signed claims (including when those claims were made) with probability --> 1.0. I'm framing this probabilistically because many sharding approaches rely on that definition. (And even non-sharded blockchains like Bitcoin itself.)
Also, I had the impression that a blockchain is a specific type of data structure.
So, what you are describing seems like a way to accomplish behavior of this kind of data structure with a different one so you may get bettee scaling out of it.
As you'll see in my other comments (and articles on it), "blockchain" is better described as a field with a set of related goals for technology artifacts, rather than a specific data structure [1]. I frame it as: it has blockchain characteristics if it's decentralized, immutable, and assets [2].
This is a much healthier framing, because it doesn't constrain the goals to a particular approach (e.g. a particular data structure).
I don't think i'm knowledgable enough to do anything other than refer you to sharding (we don't need to store the whole chain, just like in distributed computing) and the recent paper by Vitalik (Ethereum) and Joseph Poon of Lighting Networks on Plasma, which seems like MapReduce/Hadoop/Spark stuff with a blockchain
We can do a lot better than the scale of Bitcoin. And we are. Scale is part of the point. And you don't need to centralize to get scale. We did improve upon the Model T, didn't we? Or, I remember programming with 16K memory on my computer. Technology improves. And it is here too. That's what we do at BigchainDB.
Hmm, after skimming the white paper, it seems like what you are doing is you are having your consensus algorithm be that Nodes simply vote on what they believe to be the current blockchain.
How does this solution respond to someone spinning up a thousand nodes, and simply voting for their double spend attack?
In part of the paper it is states that " In a BigchainDB network, the governing organization behind the
network controls the member list, so Sybil attacks are not an issue.", which is directly contradictory to your statement that it is decentralized.
A decentralized network has no "governing organization".
> How does this solution respond to someone spinning up a thousand nodes, and simply voting for their double spend attack?
This is the classic "Sybil attack". But I bet you knew that:)
If you have a member list (ie list of public keys) of who can be server nodes, then you can control this. Each member (public key) only gets one vote. So even if that person makes 1000 copies, it's only 1 vote total from that member.
> governing organization behind the network controls the member list, so Sybil attacks are not an issue.", which is directly contradictory to your statement that it is decentralized.
A decentralized network has no "governing organization".
Great question. However the control of this organization is decentralized too. Here's how. IPDB is the BigchainDB public net, and foundation to help govern. Net: each server node is run by a "caretaker". Foundation: each caretaker has one vote. They vote to control the member list (list of caretakers), as well as IPDB board. So, it's decentralized: no single entity is controlling it.
There are other ways to curate "member lists" to address Sybil attacks. E.g. Bitcoin's PoW is basically "one electron one vote" on average (assuming everyone has a modern ASIC). In search of block rewards, many players work hard to maximize their electron spend (ie big ASIC farms), which of course eats a lot of power. Or BitShares' PoS is a riff on "one token one vote". There are more. We simplified the problem for IPDB: start with a great initial member list of reputable orgs that deeply care about the future of the internet (Internet Archive, Open Media Foundation, COALA, etc); and give them control from there. Some heavy lifting up-front to set this up allows great gains in efficiency.
Hello, just wanted to thank you for answering to every question this post gets. It really helps to get the gist of what BigchainDB is! Will read the primer/white paper for sure!
Also, I encourage you to try out BigchainDB. Within just a few seconds you can send your first transaction to IPDB (BigchainDB public net): https://www.bigchaindb.com/getstarted/
Correct, traditional blockchains like Bitcoin aren't scalable.
The whole point of BigchainDB is to bring scale to (the database part) of the blockchain space, using the learnings from distributed databases which do scale.
More info at www.bigchaindb.com, I encourage you to have a read:)
Hi, traditional "blockchains" have full replication which of course doesn't scale as you mention. Replication factor of 5000 is overkill (basically what Bitcoin has). Full replication isn't a prerequisite to building something useful. Instead, replicate less, use the extra nodes to store other shards of data, and make sure you have the right guarantees. That's the path.
Summary: Python isn't the bottleneck yet, and if it becomes one, C will become the last 1%.
I've been working on production apps in Python since 2002, including ones doing large-scale compute running 1000+ machines at once. How: 99% python, 1% C. But the trick is, you only build in the C once you've worked out all the kinks and optimized the big-picture stuff elsewhere. Python is great for not only connecting things, but rapidly iterating on algorithms and building maintainable code.
The AI / ML community has discovered this too: Python is now the most popular language in that community. Despite the heavy compute. How: most of the popular libraries have efficient C (etc) implementations under the hood.
This is exactly the philosophy we've been following at BigchainDB, with success. Python to connect things, iterate quickly in improving algorithms, and ship maintainable code. We haven't got far enough to resort to building our own C libraries yet, though many 3rd party libraries we use are implemented in C.
[EDIT] Based on the comments below, I'll now mention here too: BigchainDB wraps MongoDB, which is written in C++. And, Python 3.5+ (which BDB uses) has gradual typing, which brings many benefits of static typing to Python.
Trent, dean here. met you at ICCAD few years ago (2014?) and we had dinner with Bill Swartz at an Asian place. We discussed bitcoin and I remember it was $600 back then. Funny that it's now 6k$ and I didn't know you're gonna be in this space. Just wanna shout out that bigchaindb looks legit/cool and good luck in the future!!
Oh cool! Thanks for the ping, Dean. I hope you're well. Yeah blockchain space is a bit crazier than EDA, but some of us are engineers bringing our discipline and experience to this new field:)
Will definitely kick the tires on this one. And people should take notice that you guys didn't raise funds through an ICO. If anything, that right there is a big sign that Bigchain db is a cut above the rest in terms of riding the block chain hype train.gl
Another hi from the past! I'm going to have to check this out, since, well, it's you and there's been some discussion about "blockchain applications that aren't just trading currency" at work. Cheers! Glad to see you're still working on crazy interesting shit!
Hey Tony, great to hear from you! Yes please do check this out, we think it's interesting too:) Also interesting is BigchainDB's sister project www.oceanprotocol.com. Cheers!
To me, "blockchain-like" is about getting the benefits of decentralized, immutability (tamper-resistance), and assets.
How one goes about getting those characteristics is wide open. Most blockchain systems do have a full copy of the database at each server node, i.e. fully replicated. Also, they are "peer to peer" which means there is no distinction between clients and servers. (They do have SPV wallets though which is kinda similar.)
BigchainDB's focus has always been about scale. We're partly there but not fully: we are currently fully replicated but are targeting sharding to address that [1]. Where we do get scale already is properly distinguishing between between clients and servers. Servers are "super peers", decentralized among themselves. They do the heavy lifting, i.e storage. Apps don't need to run a server node; instead they simply are clients to the network, and of course can query >1 node.
I totally acknowledge that there are pros and cons for both dynamically and statically typed languages. (We could start a 10 page discussion here. But I've been there before, perhaps you have too, how about we save our energy? ;)
For the latest and greatest 10 pages about that; just search for the the Hickey 10 year Clojure (rant-ish in places) talk and the Haskell community responding to it.
Python has optional typing support in the standard library since version 3.5. Granted, not type _checking_, but that is offloaded to external tools like mypy[0].
I feel that gradual typing offers the best of both worlds: rapid prototyping, but also the strictness of types where you need it.
Perhaps you're looking for an append-only logging / messaging system like Apache Kafka? Good to explore what's out there and understand what's possible.
BTW using BigchainDB can feel pretty lightweight: it feels like a DBaaS but you don't have to set up the back end, you just get going. In the following, you'll have a tx on the BigchainDB public net (IPDB) in seconds. And the JS or py code to do it yourself is right there too. https://www.bigchaindb.com/getstarted/
For "why would I want to use this over a SQL alternative" see [2].
For "what kind of applications can I build with this?" see [1][2][3][4].
Cheers!
[1] https://www.bigchaindb.com/usecases/
[2] https://blog.bigchaindb.com/three-blockchain-benefits-ae3a2a...
[3] https://blog.bigchaindb.com/six-blockchain-application-verti...
[4] https://blog.bigchaindb.com/where-does-blockchain-scalabilit...