If OpenShift didn’t do the heavy lifting of deploying and securing Elasticsearch I wouldn’t be using it at all, and because of that mess I actually use Graylog in my lab at home because it’s substantially less of a pain in the ass and security isn’t a feature locked behind a proprietary license or writing your own proxy.
Graylog is exactly what I was reaching to go with (I've had a similar experience and was blown away and delighted by how easy it was to use) with but it's a bit heavy weight -- they (supiciously) don't have min requirements anywhere, the only scrap i could find was on the graylog open stack docs where they suggest you have 4GB of RAM free.
The machine I'm running on isn't small, I have the memory, but it just feels like a slipperly slope.
Also, Graylog = Java + Mongo + ES and I'm almost philosophically opposed to using Mongo for personal reasons (this is a personal project so I can afford to have some self-defeating bias).
The prebuilt Graylog virtual machine appliance (OVA) defaults to a pitiful amount of RAM (I think 512MB? 1GB?) and we used it in production successfully for a very long time in this configuration. We bumped it up but just because it seemed like a good idea, not because the memory was giving us any trouble. From our Graylog dashboard currently:
> The JVM is using 637.8MB of 980.1MB heap space and will not attempt to use more than 1.4GB
It also defaulted to a single vCPU, which seemed to be fine. It seems like Graylog can scale down pretty well if needed.
Does this include what you're giving Mongo + ElasticSearch? The graylog process isn't all I'm worried about, it's kind of the combination of the three.
Regardless, I'm probably going to just use Graylog then -- I'm not running a large environment by any means, and while I've been at a company where graylog was used in production (which is where I heard about it), people often complained about it hogging resources. Time has passed, and I'm sure that if it's good enough for you, it's more than good enough for me (especially since I'm not running anything "in production").
I still want to get the EFKK stack up and running though, right now there's basicaly two choices, ELK/EFK or Graylog or some hosted option (splunk, sumologic?, others), I'd like to at least stand up both choices once and get a feel for them (and I've done Graylog before).
Splunk’s not a bad piece of software, I just prefer open source options before proprietary solutions where feasible (which is why I don’t use EFK, I refuse to pay money for security and I think it’s bullshit that Elastic has made that part of their business model with the xpack) but for small environments the free version can get you far.
Not in any way affiliated with Elastic but XPack is now included in Elastic by default, so there's that -- of course it does say something that they included it in their enterprise offering first.
Same here on the open-source-first mentality. I also managed to get the EFK stack working so now I don't feel bad actually choosing Graylog in the long run.
Not all of the xpack features are free, security still requires a gold subscription with Elastic. In fact, there’s very little functionality in the xpac that DOESN’T require at least a gold subscription.
Graylog doesn't require tons of memory in my experience, it always benefits from more as your logs grow - but that's just a fact of life when it comes to any kind of database. I've run it on 2GB of RAM before (this is just the smallest amount I ever give a VM because that's what it takes to netinstall CentOS 7 these days) without issue on smaller amounts of logs (10-20MB/day).
I'm not a fan of MongoDB myself, but Graylog uses it as not much more than a distributed configuration store so I just begrudgingly accept it.
I don’t have statistics as it was used in my lab at home which I have recently torn down and begun rebuilding (new servers, new hypervisor and not enough of a crap given to v2v the VM’s I had instead of reinstalling).
From memory though, MongoDB didn’t use much since it mostly stored configuration for Graylog, the Graylog processes themselves took up a couple hundred MB and elasticsearch ate up everything I allowed it to (typical behavior of a database though).
I didn’t bother tweaking any of the settings and just relied on memory pressure of the VM everything ran on to limit resource usage, if you’re keeping lots of history and need fast access to it then obviously you’d need to give ES more RAM to work with.
Thanks for sharing -- I wasn't aware that Graylog only used mongo for the configuration information -- sounds like they're using it as a synchronization option... Wonder if they're working on any alternatives like etcd or even kubernetes-native synchronization options... After a little looking it looks like the answer is "no" (https://community.graylog.org/t/will-mongodb-ever-be-replace...)
If OpenShift didn’t do the heavy lifting of deploying and securing Elasticsearch I wouldn’t be using it at all, and because of that mess I actually use Graylog in my lab at home because it’s substantially less of a pain in the ass and security isn’t a feature locked behind a proprietary license or writing your own proxy.