I am currently writing a Golang client for Elasticsearch which uses the native binary protocol and I have to say the lack of documentation about it is making the process really painful!
I tried to use the Elasticsearch thrift plugin but unfortunately it does not work for the version 1.1 and 1.2
So basically I have to inspect each and every byte of each and every request and response in order to be able to send or parse data.
While developing the client a managed several time to crash the Elasticsearch server by sending malformed packets. In addition, this, brought me to review the networking part of Elasticsearch code and I think it needs a refactoring and a better, deeper and cleaner usage of Netty.
I hope they will soon sort out this and the problems mentioned in the article, since I think that Elasticsearch is really an amazing product!
This brought me to dig deeper into Elasticsearch code, find out more about its code quality, deal with machine endiannes, deal with byte shifting, think how to structure code in Golang and overall enjoy the feeling of touching the bare metal again...
I guess bc theoretically someone is paying you to write applications at a fair clip. Granted, we have no idea what your role is or what your goals of the project are, so we are probably completely wrong at your own role! :)
So if Elasticsearch is a stringified (JSON/HTTP) wrapper around Lucene with simplified setup for the web app crowd, you are making a binary-fied wrapper around it. Why not skip ES altogether, or use the JSON api?
ElasticSearch is a lot more than just a "stringified wrapper around Lucene". Lucene is used for the underlying inverted indexes, the item store and tokenization/analysis, and that's pretty much it. ES adds clustering, a query DSL, configuration, data mapping system, "river" functionality, HTTP API etc.
The clients actually acts as a cluster node and therefore has knowledge about the cluster state, its indexes and shards, because it receives notifications from it, once it joins.
This allows to execute operations on a specific shard of a specific index on a specific node of the cluster resulting in better performance than going through the HTTP interface.
It can be used to efficiently store big quantities of data, for instance logs, which then can be visualized with Kibana.
It's just unfortunate that Elasticsearch presents the problems mentioned in the article and which I also experience in production, because it has a series of plugins which makes it a good solution for specific use cases.
I'd recommend not using the native binary protocol unless you have proof it makes a substantial difference for your application.
If you need to do bulk work, connection pooling, keep-alive, and batching on the client-side over HTTP can easily vastly exceed what ES cluster can handle. Users of my library have confirmed this.
You could use my library as a guide to the abstract data types, even I don't use the native protocol.
Looking for a company where code simplicity and performance matter! I do not like to waste CPUs cycles, memory and I/O in general.
I have more than 13 years experience as a developer and I have been designing and implementing scalable, distributed and resilient systems and software, to handle over 40 million unique users per month, for the last 7 years. I am proficient with Scala, Java, Netty and Akka and I really enjoy coding in Golang. I have a strong passion for cryptography and IT security in general.
On my free time, I am working on a Golang Elasticsearch client which uses the tcp transport.
I am an open minded, friendly and business driven person who loves technology. So for any questions drop me an email !
I am passionate about cryptography and IT security and I am looking for an opportunity to work on open source projects preferably in crypto and security in general.
I have also more than 5 years of experience in scaling web sites and native applications stack with more than 50 billion page views per month.
I love talking to others, so for any question feel free to ping me anytime!
In general you should prefer crypto constructions which are a result of global competitions.
For example AES and SHA3.
You should avoid at all costs anything that has been standardized by NIST without going through years of reviews by international cryptographers. Dual_EC_DRBG is a clear example of crypto construction which falls into this category.
This is my general rule of thumb.
However knowing which ciphers one should use is not enough! You absolutely need to know HOW to use them.
A basic and superficial example is AES in ECB mode, which is semantically secure as long as you use a key to encrypt one and only one single block.
Another one is, for example, after how many encrypted blocks a key should be rotated, based on the underlying cipher used.
Once you have learnt how to use the basic building blocks of crypto you are then NOT supposed to write your own implementation and instead use existing ones....there is a small problem with this....they are broken or they either not implement all the necessary crypto constructions you need.
OpenSSL is an example of broken crypto implementation, and instead NaCl does not have TLS implemented.
So this is a short summary and my personal opinion of why crypto is hard.
On top of all this there are not enough experts out there which have the time to review crypto implementations or new and old constructions, and we are living a historical period where we desperately need crypto to protect our privacy.
> The judges who chose AES and SHA-3 as the "winners" of the global competitions are the NSA.
Sure, however this process creates alternatives and if the crypto community thinks the winner is backdoored I am pretty sure we will know it and additionally we will have a valid alternative ready to be implemented.
Additionally if the NSA/NIST modifies the specs for the crypto construction there is still the possibility to implement the original one. See SHA-3 for instance. It was about to be weakened, but the crypto community could still implement the original spec.
> That would include AES and SHA-3.
You cut the rest of the sentence and therefore changed completely the whole meaning. My original sentence included: "...without going through years of reviews by international cryptographers."
Take a look at this video of D.J.B.: https://www.youtube.com/watch?v=G-TM9ubxKIg
He makes a great example with the Dual_EC_DRBG, where many cryptographers told NIST that there could be a backdoor. NIST answer basically was: sorry too late, it has already been implemented !
So in other words, in case of Dual_EC_DRBG the standardization process was all in reverse. First NIST standardized it and then the crypto community started to review it and found problems.
