In terms of AWS costs, running your own cluster will cost significantly less, specially if you use reserved instances. The AWS ES instances cost ~1.6x more than their regular counterparts. Also, you can't reserve ES instances.
That's if you have a qualified sysadmin on-staff to manage it. We tried managing our own ES setup for a little over a year before giving up. Despite our simple needs and low traffic volume we regularly had difficult-to-diagnose issues with memory usage, node interconnectivity, and performance. With 3 EC2 nodes in a cluster we would occasionally have the cluster health go to yellow and remain in a 'recovering' sate indefinitely, bringing performance to a halt, and we'd essentially have to rebuild and swap the cluster.
Granted, none of us were proper ES admins, but we had a lot of experience working with system administration and specifically database performance and clustering. Despite that we were definitely in over our heads with ES.
Qualified sysadmin/devops here: You can run a small Elasticsearch cluster (<9 nodes) without a dedicated ops person if necessary. Run with more than 3 nodes, ensure you have a proper number of index shards/replicas, and ALWAYS use an odd number of nodes.
Some overprovisioning will be required, but with the extra infra spend you're delaying the need for a dedicated role to manage it.
Software I have to 3x over provision to keep alive is crappy software.
ElasticSearch is by far one of the most obnoxious software programs I ever have the misfortune to administer. I now avoid self hosting it at all costs.
You don't "3x overprovision" Elasticsearch in particular, you should always provision a minimum of 3 nodes for any highly-available clustered DB. 1 node may fail, you're down. With 2 nodes you can develop a partition or lose a node and neither node can elect itself master, you're down. With 3 nodes, you can lose a node or develop a partition, the other 2 nodes can reach a quorum on a new master and continue operating.
This is how it should work, in theory. However, in practice, with ES whenever a single node went down the whole cluster would fail. Trying to add a new member to the cluster never worked, nor did trying to recover the failed node, hence the cluster-swap.
It doesn't help that their logging (at least pre v2) is incredibly dense.
ES clusters will fail on the loss of a single node if you aren't running any replicas on your shards, but that's not really ES's fault. I've occasionally had a node just wig out and need to be restarted, but that's like a once-a-year thing, and I'm working on a ~TB cluster that processes a ridiculous number of writes - this isn't an underworked cluster by any means. As long as your cluster discovery mechanism is set up properly, adding and removing nodes from the cluster is about as easy as it gets. I'm certainly not saying that your experience wasn't valid, but my own experience with it has been that it's remarkably easy to manage.
ES was pretty brittle in the pre-1.x days, but from 1.0 onward it's quite easy to work with. The logging is dense, but that's because it's thorough - a feature I really quite appreciate.
You want an uneven number of master eligible nodes, so you can form a proper quorum (two out of three, three out of five). The total number of nodes doesn't actually matter. Smaller installation often make the same node master eligible and data holding at the same time, so in that case you'd prefer an uneven number of nodes. Once you move to a setup with dedicated master eligible nodes, you're freed from that restriction. You could also run 4 nodes with a quorum of 3, but that will make the cluster unavailable if any two nodes die. The worst setup is 2 nodes since you can only safely run with a quorum of 2, so if a node dies you're unavailable.
I've been managing out elasticsearch cluster for the past year and a bit. It grew from a single node that also ran kibana, logstash, and nginx, and stored our mysql backups to 9 data nodes, 1 client node, and a dedicated master. I have faced issues, but never had to rebuild the cluster. For the most part, reading up on the ES docs, and making a config change fixed the issue. Sometimes I've had to restart a node, but thats rare.
At a previous company, we had a culture of sending out prank emails from open machines.
I once sent out an email from a co-founders account which said that he was fed up with the crappy codebase and was hiring a new team to rewrite it from scratch and that the other (non-tech) co-founder was to take over the existing tech team. No one took it seriously (non-tech co-founder helped me draft the email, the other one laughed while reading it after the fact), but there was a board member in the mailing group who thought it was serious and started sending panicky emails to the co-founders.
I'd go to the bookstore and leaf through a few popular science books and get the ones that I liked the most.
Other than that, there are a lot of TV series and youtube videos/channels that do an excellent job of explaining science and maths. Dr Brian Cox does a brilliant job in his Wonders of the Universe,and Wonders of the Solar System series.
That's quiet a high salary you currently get. Usually when foreign companies hire developers in India, they do it to reduce costs(for what you get, your company could easily hire 5 really good developers, or 10 good ones). I'm guessing that your company believes that compensation should be based on what they believe the developer is worth and what they are willing to pay, instead of where the developer lives.
$250k/year is probably twice the median salary for a developer in SF. Yes it's not quiet the same as it would be in India, but still is pretty good. I personally would take it, just because I feel it's a better place to live and work.
To put it into perspective, Google pays it's CFO a base salary of $650K a year. Of course there are bonuses, and stock options that make it much more than that, but I doubt you'll get anywhere close to what the converter says.
@elssar: Yes, my company actually pays based on what the developer is worth and pays the same amount for same work across the world.
>I personally would take it, just because I feel it's a better place to live and work.
This is what is vexing me. I feel the same but I know that my family would need to make some pretty big lifestyle changes if I were to move there.
Thanks for the Google's CFO base salary number. That puts the converter out of equation now.And thanks for your overall perspective too.
EDIT: The salaries in product companies in India have significantly gone up in the last year and a half with so many FIIs pouring in funds. So, I think $200K would only get 2 good developers at my experience level now.
> my family would need to make some pretty big lifestyle changes
That is something I don't have to consider. That makes the decision harder. I know someone who chose to work on his own startup in India, over moving to Mountain View, because he didn't want to move his family there.
I disagree: Given the specifics of the situation, I'd say that going to SF is a huge risk and the money dot not justify the adjustments he will have to make (culturally and financially).
Of course everything depends on the details (age, marital status, aims, etc.).
