I would suspect the poster has already weighed the cost vs. benefit of writing efficient code. Many domains do require careful attention to performance.
For most American dialects, the sounds are nearly identical, but not quite: at the beginning of a syllable, the "r" is an approximant (a type of consonant that starts with an emphasized vowel sound), rather like "y" or "w", whereas at the end it is a true vowel, again like "ee" or "oo" respectively.
It's not as much data as one might think, especially if you store the deltas -- there's not that much variation between individuals. That fact has actually been exploited to a great extent in enabling the $1,000 genome.
How much is "not much?" How many "values?" If stored as byte strings delimited by \n, how many lines? (I don't know if these are even answerable questions at this point, just wondering.)
The raw data (the original image files), while mostly transient today, is massive. At some of the big genome centers they are in the double digit TB's of data processing per day
One can easily estimate the cost of a query in a relational database via EXPLAIN.
What relational databases do is decouple such implementation details from semantics, enabling higher levels of abstraction and therefore reasoning. Similarly, I'm happy to abstract the low-levels details of dictionaries in Python and not code my own hash table from scratch, or to use a compiler and not write assembly language. And while there are inevitable corner cases where hand-coded logic will outperform what a system can guess, handicapping yourself for a few extra cycles is something routinely (and rightly) derided in most programming circles.
Nobody debates that NoSQL systems are great for hand-tuning corner cases like these. It's the idea that they are a viable alternative to the relational model that is so objectionable, as it's a step backwards in abstraction.
What bothers me most about the NoSQL movement is that it seems to be in conflict with a central tenet of our community: to avoid premature optimization, especially when at the expense of of expressiveness or correctness.
While normalized relational databases are indeed slower than key/value stores, they also foster data integrity and the ability to compose arbitrarily complex queries with little effort (assuming one actually groks SQL). One can always optimize trouble spots in a relational database later on (possible even with a KVS), but using a KVS from the get-go is bound to lead to pain.
If we happily use Ruby, Python, et al. while chanting the mantra of premature optimization being bad (and rightly so!), why does this not apply also to databases?
That's like saying a Formula 1 team should first design a Ford Focus and then optimize it to win the world championship. It just doesn't make sense to call that process an optimization. The same is true for hi-performance/hi-volume use-cases where KV stores make sense.
Also, putting the optimization issue aside, the KV data access model is actually quite natural for web applications. This is hard to believe (seriously), and you have to see it to believe it. Note that in web apps, being able for the administrator to go in and issue complex queries/updates is much less important than for eg. a billing system. I use an RDBMS for an old billing project, and it wouldn't make sense to use a KV store, but I use a KV store for a web app I'm writing, and it turns out to be a great fit, more natural than an RDBMS.
Sure, I suppose it matters a lot on the type of Web application you're building. If you have a relatively simple domain model with very few relationships, I'm sure a KVS works pretty well. Still, for anything other than applications with simple domain models, or ones that need tremendous performance, a KVS is inappropriate outside of isolated optimization -- in practice, one runs into headaches with non-normalized data very quickly (think MBAs and their giant, messy Excel spreadsheet).
In most web apps data in per-user (or per-object, whatever that object is), and it's not a major headache to denormalize it to be per-user. One notable is exception is a social network, but that can be handled with some care.
We're talking about web apps written by programmers, not MBAs. Sure, if your web apps are written by MBAs, then they should use an RDBMS, because they can do less damage (?).
I'm not arguing against RDBMS/SQL here, all I'm saying is that a KV store is actually a good choice for the domain of web applications. You can also use an RDBMS of course, as we have for the last 10 years with the LAMP stack, as long as you're handling a managable amount of traffic. Even with a lot of traffic you can handle it with an RDBMS, it just stops making sense at some point.
The NoSQL movement is in direct support of the central tenet: do the simplest thing that could possibly work.
Relational databases are complex beasts that have a lot more configuration and administration overhead. They are overkill for most use cases. It's not just about scale, it's about simply wanting to be able to store and retrieve values because that's what we do more than anything else. Key-value stores are simple... you put things in and get things out. Instead of calculating things on read, you calculate and store them on write.
Key-value stores make a lot of things braindead simple that are otherwise fairly complex in relational databases.
Not just uninformative, but misleading too: there are many relational query languages besides SQL (e.g. Tutorial D). Is the conference about non-relational database technologies, or any alternative to SQL?
Hah, I threw LINQ in there figuring something like D would be too confusing/obscure. But your reminder sent me a-googling - wikipedia opens with:
"D is a set of requirements proposed by Christopher J. Date and Hugh Darwen in their book The Third Manifesto "
So maybe NoSQL people need a movement after all. The relational people are at _least_ on their third manifesto. Lot of catching up to do!
No, i missed the that they were talking about kilograms not grams. Once again my "no posting on the internet before coffee" rule shows its wisdom... and me my foolishness for ignoring it. Good spot.
This is generally the case across the physical and life sciences too. Even "revolutionary" fields like epigenetics and genomics are effectively refinements and extensions of the pioneering work done in molecular biology during the 1960s and 70s. The same goes for particle physics, where the framework for the standard model is decades old. It seems that scientific innovation occurs in punctuated equilibria.
There's no question things go in S-curves, but I really get the feeling that the kind of people who did great work in the 60s and 70s generally aren't allowed to go into academia today. I personally would probably get a PhD in another life, but there's no point because academia is generally designed to filter people like me out, so I would have to spend 2-4 years convincing the right people to go to bat for me and it's just not worth it.
I actually abandoned a PhD (after 2+ years) in favor of industry. The modern academic world -- at least in CS -- seems largely to consist of producing as many papers as you can, putting forth unimportant results, so as to get more 'points' and another funded project and more students. (British universities really are assessed on published papers, with different weightings accorded to international conferences and journals. It's a game.)
To do anything interesting means reducing or delaying your paper output (which your supervisor will not appreciate), and to get the results published requires you to write so as to persuade people with no vested interest in publishing novel output to agree with you... but without being able to have a dialogue.
I recall one amusing paper review by three reviewers which was essentially
1. "the treatment of X is great, but I'd like to see more Y",
2. "too much Y, not enough X",
3. "both X and Y are fine, but I'd have been really interested in seeing Z instead", where Z was entirely unrelated.
How on earth could an author anticipate the preferences of those authors and get three recommendations? Well, picking a non-controversial subject would be a good start.
Multiply this anecdote up to the size of the academic and industrial systems, and it's no wonder that so many people decide that academia is not for them. These people tend to be lumped in with the people "not cut out for it", but I think a significant percentage simply realize that academia has a different value system, and don't want to participate. Some of those people go on to do interesting or revolutionary things in corporate research, or even in mainstream industry.
Would the world be better if they were in universities instead? Probably not... but we'd have our interesting scientists back.
Academia does tend to have a different value system, but it's a mistake to suggest that the one used by industry is better. It's just different.
Yes, in academics, professors X,Y and Z will usually try to get you to introduce their pet subjects in review. You also later find out that getting a paper approved has a lot to do with who you know (as opposed to what you know), and that getting the choice talks, collaborations and grants has a lot more to do with schmoozing and networking than anyone likes to admit. It's a social problem.
But you know what? That's life. Humans are fundamentally just a bunch of howler monkeys, flinging poo at the cage walls and trying to steal the shiny rock (or the girl) from the alpha. Put more than a few people together in any activity, and you'll get competition, preening, back-biting and politics.
In academics, powerful professors demand that you brown-nose and compete for favor; in industry, powerful rich people do the same. In academics, you find that most days are spent managing mundane chaos, with real research advancements happening only a few times a year; in industry, most of your time is spent appeasing people, reacting to short-term crises, and coordinating with your co-workers. When you really think about how chaotic humans are in groups, it's amazing that forward momentum happens at all.
Point is, I don't think it's fair to imply that "interesting" scientists migrate to industry because of some fundamental difference in values. Personally, I made the migration because the monetary up-side is much better in industry. Otherwise, I think the bullshit-to-accomplishment ratio is roughly the same.
What I mean is that the best grad schools in the US are purposely designed to have an extreme bias toward selecting students with the best grades and the highest GRE scores. You can still get in without good grades and without taking the GRE if you've done very good research and have earned the trust of the right people, but it's very difficult. And even if you do get accepted, you'd probably face an uphill battle throughout your entire career.
This is true at all the top CS programs as well. They look at:
1. Past research (including published papers)
2. Recommendation letters
GRE scores are a pre-filter (not set too high, I think, and certainly the CS subject GRE is ignored, even when it is required), grades matter only slightly, and the admission essays are basically a chance to disqualify yourself by saying something stupid.
I've heard this too, but do the admission statistics actually confirm it? I suspect it's more along the lines of, "we only care about your recommendations, as long as you have above a 3.7 GPA." I could be wrong...
