> you do have good reason to believe that the choice is harmless.
The issue you will run into here is that 95% confidence means that you will only have a false positive 5% of the time. It does not mean a neutral finding is 95% likely to be neutral. The lever that controls that is statistical power, which is oft-ignored in conversations about A/B testing. Most statisticians use 80% power, which means a full 20% of neutral findings were false negatives.
That is very true. However, you do have much better reason to believe that the option is neutral than you have to believe that the option is beneficial. In an example like the one given in the article, you also likely have enough statistical power to be reasonably confident that the option is close to neutral, so if you're making a negative decision, it's probably not strongly negative.
Yeah, that's a neat project. I was already thinking about building this before they launched, and their positive reception spurred me on.
If you're already using Heroku Postgres there's a ton of overlap between the services, but I'm hoping to provide a decent open source alternative for the rest of us. :)
If you can run Rack apps in your environment, it would be pretty straightfoward to add a SQL Server driver to Oculus. Right now it supports MySQL and Postgres.
It's designed to be used within a team environment, where you trust everyone. (Although even then, I highly recommend using it with a readonly SQL account!) I would be skeptical of any situation where you could run raw untrusted SQL without any risks, at least without reimplementing MySQL's AST parser.
As for the demo, it's isolated, and locked down as well as MySQL will allow. (For starters, just SELECT privileges.)
Author here. I do a lot of analysis each day and am constantly sharing it with my team by taking screenshots of Sequel Pro. I built Oculus so I could keep a centrally available repository of my research, that everyone could access and collaborate on. It definitely has a ways to go, but I would love feedback!
Also, this is my first attempt at providing a live demo of software that can execute arbitrary SQL. I'm probably crazy for doing this, but I thought it was a lot more interesting than doing a video. If the demo suddenly disappears, you'll know I wasn't able to harden it enough... :)
This is an interesting study, but I found the HN title to be misleading. Here are the bullets from the concluding paragraph:
* The American public has long held generally positive attitudes toward the space program, but is not very familiar with its details.
* Over the history of the space age, an average of more than 60 percent of those polled rated the job done by NASA as either ‘‘excellent’’ or ‘‘good.’’
* Most Americans have shown support for space exploration and view it as important over the years, but also believe that federal money could be better spent on other programs.
* Most are also in favor of NASA as an organization, but are relatively unfamiliar with the majority of its activities and objectives.
* These polls also suggest historically close relationships between public perceptions of NASA and spaceflight depictions in popular culture, especially film. These images from popular culture, coupledwith real-world accomplishments in spaceflight, work together to create powerful visions affecting the public consciousness.
This is a powerful approach when you can quantify your regret. For many startups, however, it's important to understand the tradeoffs involved in moving one metric upward or downward. To take Zynga as an example, they care about virality at least as much as engagement (or perhaps moreso). Adding or removing a friendspam dialog is likely to trade some virality for user experience. What percentages make or break the decision? Sometimes this is a qualitative call.
In environments where you need to look at the impact of your experiments across multiple variables, and make a subjective call about the tradeoffs, it's really important to have statistical confidence in the movement of each variable you're evaluating. This is a key strength of the traditional A/B testing approach.
I love Giles. His first Archaeopteryx talk was one of the best Ruby-related talks of all time. He's way too harsh on Rails though. The original "blog in 15 minutes" demos that catapulted Rails into the limelight are still possible with Rails 3. Things like 'gem install rails' still work. If you are determined to skip bundler, you can do that too. What's more, unlike 2005, in the 15 minutes it takes to build a Rails blog, you can deploy to Heroku and have a live site running.
Node is fun. V8 smokes MRI. But Node's web framework ecosystem has a long, long, long way to go before one can reasonably say that Rails is a bad investment. If anything, we should be celebrating the number of options that are open to us. It's a great time to be developing for the web. Good things will come from Node's ascendancy, whether its frameworks dethrone Rails or not. The negativity is unnecessary.
I'd go further and say, that it will be very hard to have a good MVC framework with Node before the async callback mess is brought under control. Maybe we'll even need something like Iced CoffeeScript for this.
I'd caution against any "I'd go further to say" statements. Either you know because you've tried, or you don't and are simply guessing.
As someone who has made an MVC framework in Node.js and launched a production site using that framework, I can assure you that it's not hard to avoid async callback mess. I can also assure you that many others have had success doing this.
It's really not difficult to write beautiful, modular, and readable code in node.js without going past 80 columns and without using Iced CoffeeScript.
I suspect that comments like this only further reinstate the theory that most people are dismissing Node primarily due to the annoying amount of hype. There are better reasons to not use Node, but imo, this isn't a very strong argument.
All the examples, github repos, books, and tutorials I'm seeing with my dive into node say exactly the opposite - the code becomes a mess of callbacks. It looks like enormous work goes into avoiding callbacks, similar to the amount of work that went into Rails in 2005 so that server-side folks could avoid writing JavaScript.
I'd love to see this framework you wrote. Is it open-source? or is it something you're unable to share?
You have to think about async code differently than you think about synchronous code. In PHP you might write:
$data = getData();
And people want to substitute the return for a callback in js.
getData(function(data) {
// Do something with data.
});
That indeed does lead to callback mess. But when you are coding in js you rarely should use anonymous functions. I mostly just use them when I might want to perform recursion. Going back to my example, I would write the operation as an object like so:
Wrapping things in objects doesn't seem to help me.
I'd still end up with this, thanks to callbacks.
app.get('/documents', function(req, res) {
Document.find().all(function(documents) {
// 'documents' will contain all of the documents returned by the query
res.send(documents.map(function(d) {
// Return a useful representation of the object that res.send() can send as JSON
return d.__doc;
}));
});
});
Seems to me that what's missing is something that lets me write it in a more simple way. What I have here looks like an absolute nightmare of coupling. So if you have some suggestions you could point me to that show a little more detail, I'd be really grateful.
I would create a DocumentRequest object that takes your req and res, then break apart your callbacks to separate functions on the DocumentRequest prototype. Then you would simply consume it with:
The async callback mess is not really a problem, there is about zillion patterns and libraries to handle it. The problem is error handling in callbacks, capturing stack traces from the async parent callers, and generally not crashing Node if you forget to try/catch your async handler. However, this problem is known, and this is being worked on in the form of "domains". Btw. Iced CoffeeScript doesn't solve this a single bit.
If you have callback hell in your code, it's not because the problem hasn't been solved, but simply that you didn't put the effort to npm install a simple library.
I agree that it is a problem if neglected, but most people don't allow it to be a problem in the first place.
It's like knocking on PHP because it could be used for templating and business logic on the same file. Just because you could, doesn't mean you have to (or should).
I honestly don't feel that it's as big of a problem as people make it out to be. It's simply a minor annoyance. The error handling is the killer.
Could you give a brief code example how monads eliminates the problem of error handling? Do you mean to put try catch in the monad around every async step it takes in? Just the API for how this would even look would be great.
this is just a sketch to give you an idea of what I am talking about.
right well you've probably used jquery before, so the api would look something like this. You start with a sequence of async actions you want to perform.
Now this is a bit different from jQuery since all the methods here do is push method names and arguments onto a stack. it's the .end() method that reads in the stack and goes through and executes it. This means that each method along the way we've constructed a partial computation value- that is, the stack, which we can pass around as a value- just leave off the "end" method...
var scrapeHN = $(ctx).geturl("http://example.com).parseExample().renderTemplate();
now we can modify the computation by turning it into an error handling monad...
var ecm = ErrorCatchingMonad(scrapeHN, errorCallback);
this wraps each method in the monad with a try catch block, and calls the callback if an error gets thrown, and stops the chain from getting executed further. in fact this is how the origial monad was created
var $ = DeferringMonad(scrapingAPI);
which takes a normal jquery style plugin interface/chaining api and turns it into that deferring stack type api with an end(); method.
real answer: It's easy and fun to whine about things you don't get in a language/paradigm you don't know. So easy and fun in fact that's almost entirely what programmers talk about to each-other. But whining hardly gets anything done, does it. The answer is: It's not a problem if you know what you're doing. And that applies to all other instances of this programmer discourse pattern.
The issue you will run into here is that 95% confidence means that you will only have a false positive 5% of the time. It does not mean a neutral finding is 95% likely to be neutral. The lever that controls that is statistical power, which is oft-ignored in conversations about A/B testing. Most statisticians use 80% power, which means a full 20% of neutral findings were false negatives.