(I work at Stripe, and helped with a lot of our early Consul rollout)
We did consider Synapse (and Nerve[1], which they used to call SmartStack when used together) when we were building out our service discovery, and went with an alternative strategy for a couple of reasons.
Even though it's not directly in the line of fire, we weren't super excited about having to run a reliable ZooKeeper deployment; and we weren't excited about using ZooKeeper clients from Ruby, since it seems like the Java clients are far and away the most battle hardened. (IIRC Synapse only supported ZooKeeper-based discovery when it was initially released)
We wanted a more dynamic service registration system than Synapse seems to be designed for. Changing the list of services with Synapse seems to require pushing out a config file change to all of your instances, which we wanted to avoid.
Our configuration also looked less like SmartStack when we first rolled it out - we were primarily focused on direct service-to-service communication without going through a load balancer, and we expected to be querying the Consul API directly. The shape of Consul felt like it better fit what we were trying to do. We've ended up diverging from that over time as we learned more about how Consul and our software interacted in production, and what we have now ended up looking more similar to SmartStack than what we started with.
There aren't a _ton_ of different ways to think about service discovery (and more broadly, service-to-service communication). One of my coworkers wrote[2] about this some in the context of Envoy[3], which also looks a lot like SmartStack. It's not terribly surprising to me that a lot of them converge over time - e.g. the insight of trading consistency for availability is key.
I think it's really interesting that "what we've already got setup" is such a big driver in which systems we pick. For example, in 2013 Yelp already had hardened Zookeeper setups and Consul didn't exist ... and when it did exist Consul was the new "oh gosh they implemented their own consensus protocol" kid on the block, so we opted for what we felt was the safer option. I do have to be honest that I was also pretty worried about the ruby ZK library, but to be totally honest it's been relatively well behaved, aside from the whole sched_yield bug [1] occasionally causing Nerves to infinite loop shutting down. We fixed that with a heartbeat and a watchdog, so not too bad. Which technologies are available at which times really drives large technical choices like this.
Consul template is undeniably useful, especially when you start integrating it with other Hashicorp products like Vault for real time rolling your SSL creds on all your distributed HAProxies. And I think that the whole Hashicorp ecosystem together is a really powerful set of free off the shelf tools that are really easy to get going with. I do think, however, that Synapse does have some important benefits, specifically around managing dynamic HAProxy configs that have to run on every host in your infra. For example, Synapse can remove dead servers ASAP through the HAProxy stats socket after getting realtime ZK push notifications rather than relying on healthchecks (in production <~10s across the fleet, which is crucial because if HAProxy healthchecked every 2s we'd kill our backend services with healthcheck storms ... because we've totally done that ...), Synapse can try to remember old servers so that temporary flakes don't result in HAProxy reloads, and it can try to spread and jitter HAProxy restarts so that the healthcheck storms have less impact, all while having flexibility in the registration backend (Synapse supports any service registry that can implement the interface [2]). However, there are some pretty cool alternative proxies to HAProxy out there and one area that Consul is really doing well on is supporting arbitrary manifestations of service registration data using Consul template; SmartStack is still playing catch up there, supporting only HAProxy and json files (with arbitrary outputs on their way in [3]).
I enjoyed the article, and thank you to the Stripe engineers for taking the time to share your production experiences! I'm excited to see folks talking about these kinds of real world production issues that you have to deal with to build reliable service discovery.
I respectfully disagree. I'm all for root cause analysis and taking the time to fix things upstream, but I also think that it's easy to say that and hard to actually do it.
Yelp doesn't make more money and our infra isn't particularly more maintainable when I invest a few weeks debugging Ruby interpreter/library bugs, especially not when there are thousands of other higher priority bugs I could be determining the root cause of and fixing.
For context, we spent a few days trying to get a reproducible test case for a proper report upstream, but the issue was so infrequent and hard to reproduce that we made the call not to pursue it further and just mitigate it. I do believe that mitigate rather than root cause is sometimes the right engineering tradeoff.
A bug like that is something that you want to squash because the cause might have other unintended consequences that you are currently un-aware of. To assume that there are no other consequences is the error, and the only way to make sure there are not is to identify the cause. This sort of wiping things under the carpet is what comes back to bite you a long time after either with corrupted data or some other consequence.
Now, given the context it doesn't matter whether or not the company or the product dies so I can see where you're coming from but in any serious enterprise that would not be tolerated, but when your code base already has 'thousands of other higher priority bugs' it's a lost cause, point taken. But at some level you have to wonder whether you have 'thousands of higher priority bugs' because there is such a cavalier attitude to fixing them in the first place.
> in any serious enterprise that would not be tolerated
I think that's a bit of a true scotsman fallacy. We use a lot of software we didn't write, and a lot of it has bugs. The languages that we write code in have bugs (e.g. Python has a fun bug where the interpreter hangs forever on import [1]; we've hit this in production many times). Software we write has bugs and scalability issues as well. We try really hard to squash them. We have design reviews, code reviews, and strive to have good unit, integration, and acceptance tests. There are still bugs.
I'm glad that there are some pieces of software that are written to such high standard that bugs are extremely rare (I think that HAProxy is a great example of such a project), but I know of very few in the real world.
I'd argue that m3 and c3 instances shouldn't be classified as "previous generation", since the m4 and c4 instances don't have any local ephemeral storage. Given the recent EBS incident in GovCloud, I think it's still pretty reasonable to be skeptical of EBS.
This actually took a fair amount of digging! We've been using some version of unilog for over 4 years now (longer than I've been at Stripe), and we'd mostly forgotten why we switched. What follows is more the result of historical exploration and guesswork than authoritative statement of original truth.
I'm fairly confident that the impetus for unilog was timestamp prefixes for our log lines. We wanted timestamps (so that we weren't dependent on all applications adding them). multilog is capable of doing writing out timestamps, but it formats them with TAI64N. We wanted something more human-parseable.
Once we had it, we started adding other features. These days, I'd say the most useful thing unilog does for us is buffer log lines in memory. We would occasionally see disk writes on EC2 hang for long enough that the in-kernel (64k) pipe buffer would fill up and cause applications to stall.
And an update! I talked with the engineer that wrote unilog originally.
The original headline feature of unilog was that it wouldn't block writes if the disk filled up. multilog does - if it can't write a line to disk, it stops ingesting data off of stdin, which eventually causes the application to hang writing to stdout.
unilog sends you an email and starts dropping log lines, which we decided better matched the tradeoffs we wanted to make - losing logs sucks, but not as much as blocking your application until you figure out how to free up disk.
Wow--thanks for the update. Buffering to memory before writing to disk is definitely a great feature (especially when operating out of AWS) and could be enough for us to switch to unilog.
Regarding the timestamps. We've actually gotten quite used to TAI64N--it's definitely not human parseable but it is extremely specific :) We end up just piping logs through tai64nlocal or converting it at the logstash stage.
Take a tcpdump and open it in Wireshark - you can't see the content of the requests, but the first packet will probably have an SNI header that tells you what hostname you're trying to connect to.
Have you looked at einhorn? We've been running HAProxy under einhorn for a while now, using something like https://gist.github.com/ebroder/36b2f4f3aa210b9d9f3d to translate between HAProxy's signalling mechanisms and einhorn's signalling mechanisms.
Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.
For Stripe's webhooks, we currently use the mechanism described in this post (resolve the IP address, munge the URL we're connecting to, and manually add a Host header), but we've run into a lot of problems.
If your client library supports SNI, you're likely going to send the wrong SNI hostname (we had to disable Ruby's SNI feature because sending an IP address in the SNI field broke a handful of sites), and cert verification generally becomes quite tricky.
We're in the process now of switching to a simple go-based HTTP proxy, which tries to pass through connections as faithfully as possible, but rejects connections to internal IP addresses. Here's the current implementation (though I'm still playing with it): https://gist.github.com/ebroder/ae9299e0078094211bde
This turns out to be way simpler - all HTTP client libraries have good support for proxies, and it doesn't interfere with anything about the SSL handshake (since HTTPS connections go through a CONNECT-based proxy).
We also looked at using Squid or Apache, but concluded that they were much heavier weight than we needed and it was difficult to make them behave as transparently as we wanted (e.g. they always want to add headers and such).
Sorry about that! Has to do with some anti-fraud tools we've built into Stripe Checkout. Should be fixed now, so you shouldn't see that again, Ghostery or no. Feel free to let me know (evan@stripe.com) if you (or anyone) still sees it.
If you're getting cheaper offers from other providers, you should get in touch with us (support@stripe.com). We can do volume discounts, but we can also just help you compare prices - many other providers have cheaper sticker prices, but because they frequently have other fees, we'll often come out cheaper than you think.
We can do that too! (We can't specifically do SEK yet, although we're working on it - should have it soon)
We've tried to make all of this as easy as possible - you can associate bank accounts in different currencies with your Stripe account. If you make a charge in a currency you have a bank account for, we'll transfer it directly; otherwise we'll convert to your account's default currency and transfer it to that currency's bank account.
We done have some special needs though, so I doubt Stripe is right for us. We need a few more currencies and we expect to have the money transferred without any currency conversion ( If the customer pays in Swedish Kronor, we want the amount transferred to us in that currency ). Also we need support for at least one local card.
It's not meant as a put down or anything, I just fail so see what the big deal is with Stripe.
That's actually no longer true - as of today for UK users, we can convert USD to GBP or EUR before transferring it to you. Just create the charge with a currency of "usd", and as long as you haven't manually setup a USD bank account (which you probably haven't), everything should just work.
We use the live exchange rate at the time of the charge, and there's a 2% fee for currency conversion on top (compared to e.g. 2.5% from PayPal). You can how much you'll receive for each charge in GBP immediately afterwards.
If you have a new enough ssh client, I'd personally recommend setting ControlPersist yes.
This fixes the UI wart where your first ssh connection to a server has to stay open for the duration of all your others (or your others all get forcibly disconnected).
It's not perfect. If the name of a server changes but you already have a control socket, it'll use the socket and connect to the old server. And it also takes it a while to pick up networking changes that break your connectivity, though I've hacked around that with a script I keep running in the background (Linux only, at the moment; requires gir1.2-networkmanager-1.0):
#!/usr/bin/python
import os
from gi.repository import GLib, NMClient
def active_connections_changed(*args):
for sock in os.listdir(os.path.expanduser('~/.ssh/sockets')):
os.unlink(os.path.join(os.path.expanduser('~/.ssh/sockets'), sock))
c = NMClient.Client.new()
c.connect('notify::active-connections', active_connections_changed)
GLib.MainLoop().run()
There's some contention with my coworkers about whether ControlPersist is actually desirable given the tradeoffs, but I personally think it's a huge improvement.
We did consider Synapse (and Nerve[1], which they used to call SmartStack when used together) when we were building out our service discovery, and went with an alternative strategy for a couple of reasons.
Even though it's not directly in the line of fire, we weren't super excited about having to run a reliable ZooKeeper deployment; and we weren't excited about using ZooKeeper clients from Ruby, since it seems like the Java clients are far and away the most battle hardened. (IIRC Synapse only supported ZooKeeper-based discovery when it was initially released)
We wanted a more dynamic service registration system than Synapse seems to be designed for. Changing the list of services with Synapse seems to require pushing out a config file change to all of your instances, which we wanted to avoid.
Our configuration also looked less like SmartStack when we first rolled it out - we were primarily focused on direct service-to-service communication without going through a load balancer, and we expected to be querying the Consul API directly. The shape of Consul felt like it better fit what we were trying to do. We've ended up diverging from that over time as we learned more about how Consul and our software interacted in production, and what we have now ended up looking more similar to SmartStack than what we started with.
There aren't a _ton_ of different ways to think about service discovery (and more broadly, service-to-service communication). One of my coworkers wrote[2] about this some in the context of Envoy[3], which also looks a lot like SmartStack. It's not terribly surprising to me that a lot of them converge over time - e.g. the insight of trading consistency for availability is key.
[1] https://github.com/airbnb/nerve [2] http://lethain.com/envoy-design/ [3] https://lyft.github.io/envoy/