This is a mistake. The Apache foundation doesn't sell $250/hour consulting gigs for its primary source of revenue. Neither does the Linux Foundation, the SQLite Consortium, or other massive, mission-critical open source products.
This is the wrong funding model. It keeps money in OpenSSL developer's pockets, but there is no financial incentive for any OpenSSL developer to work on foundational improvements to OpenSSL. He said himself: there is over $100,000 in open contracts for competent developers to work on non-foundational improvements to the project. If you are an enterprising developer with good C skills and a knack for crypto projects and you apply to work for the OpenSSL foundation, are you going to start servicing that $100,000 pool of contracts or are you going to pretend that money doesn't exist and live on ramen?
If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to meet their particular needs, then none of that money is going to developers to strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing. It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else.
Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?
No one will pay an hourly rate to accomplish these tasks. Google is not going to pay by the hour for a developer to stare at a function until they grok it; they want a feature. Joe Company will not pay for developers to write unit tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know how to make their code handle it.
This model needs to go away. Competent OpenSSL developers time is too valuable to waste on client asks. Their project is too important, and as long as the money is flowing only for novel features and not structural improvement, then that money will dictate that only new features are developed.
This is one of the better comments I have seen on OpenSSL in the past week. Well said.
"This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing."
It doesn't just look like a hodgepodege of accumulated hacks, it is a hodgepodge of accumulated hacks. :)
"It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else."
One could say this with respect to many popular open source projects, including ones with corporate sponsorship. The complexity just keeps building over time and there is no such thing as "finished, accepting bug fixes only".
"Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?"
Those are rhetorical questions. We know the answers. Alas, when the people who pay for (open source) software and consulting pay to have "features" removed instead of added, "pigs will fly".
Doug McIllroy is quoted as saying, "The hero is the negative coder".
(Just in case this need explanation:
Prof. McIllroy is the mind behind UNIX pipes and one of computer science's most prominant contributors.
"Negative coder" means someone who removes code instead of constantly adding, or "committing", new code.)
We could really use some more heros. And as we switch away from OpenSSL there will be a lot of links to libssl to remove.
Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free".
My guess (and hope) is that pathological requests for "features" to be added would be met with heavy scrutiny. The authors already have day jobs in academia.
As a side note, the NaCL library you mention does only a fraction of the things OpenSSL does. OpenSSL could certainly stand to be broken into smaller components, but trying to compare it with a very small library that does mostly primitive operations is...an improper comparison.
I like Dan's work and have used in in projects, I just think your comparison and analysis are quite off base.
You are entitled to your opinion and your preferences.
As I am to mine.
From the tweetnacl.cr.yp.to paper:
"OpenSSL is the space shuttle of crypto libraries. It will get you to space, provided you have a team of people to push the ten thousand buttons required to do so. NaCL is more like an elevator -- you just press a button and it takes you there. No frills or options.
I like elevators." - Matthew D. Green, 2012
Yes, it is improper to compare a space shuttle to an elevator.
It's also absurd to use a space shuttle when all you need is an elevator.
Use whatever you want. Not everyone's needs are the same.
I like small components that are independent. The OpenSSL binary is feature for feature one fo the most complex I have ever used.
I prefer simplicity. That's just me.
Not for everybody. But some might desire it.
You have my sincere apologies for daring to mention an OpenSSL alternative.
The fact that this NaCl is so small and limited is the whole point.
I think you should reread what I said -- I think it needs to be componentized, because OpenSSL does a lot. Plus has a bunch of utilities to do things.
Comparing it to a library that is mostly crypto primitives is not a fair comparison.
Also - I'm still curious of examples of "hacks upon hacks" for my own curiosity. I've been using OpenSSL in a number of projects for 15+ years, so maybe I am used to certain things.
> Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free".
With all due respect that is complete bullshit. I do not care that you put quotes around free. Writing "free" will never be considered to include sums in the hundreds of thousands of dollars. More importantly blatant lies like this muddy the debate and set outrageous expectations. The Nacl project gives the following description of funding:
NaCl was initiated by the CACE (Computer Aided Cryptography Engineering)
project funded by the European Commission\'s Seventh Framework Programme
(FP7), contract number ICT- 2008-216499. CACE activities were organized
into several Work Packages (WPs). NaCl was the main task of CACE WP2,
\"Accelerating Secure Networking,\" led by Tanja Lange (at Technische
Universiteit Eindhoven) and Daniel J. Bernstein (at the University of
Illinois at Chicago, currently visiting Eindhoven). CACE nished at the
end of 2010 but NaCl is a continuing project.
...Many of the algorithms used in NaCl were developed as part of
Daniel J. Bernstein\'s High-Speed Cryptography project funded by
the U.S. National Science Foundation, grant number ITR-0716498.
I found the funding information for ITR-0716498. djb is listed as the PI for the project.[^1] I could only find the high level funding of ICT-2008-216499.[^2] (wtf EU?) CACE WP2 is only one component of the project. I would love it if someone with better knowledge of EU funding can find the funding for the WP2 line item. The figures are:
NSF ITR-0716498 funding: (USD) 400,000.00
EU 2008-216499 funding: (EUR) 4,733,078.00 ***NEED WP2 line item***
The tweetnacl implementation lists two more funding sources. As above it was easy to locate the NSF funding but I totally struck out for the nwo funding:
NSF 1018836 funding: (USD) $436,203.00[^3]
NWO grant 639.073.005 funding: ???????????
Don't get me wrong, I have a lot of respect for djb and I think he and his coworkers deserve every fractional euro/dollar of funding that they received but they did not work for free. Most importantly they should not be expected to work for free.
NB: This is the nwo funding site: http://www.nwo.nl/en/funding I think the english version may have a reduced set of features. I can not find the this grant information on the site.
No, "more or less for free" is not close to hundreds of thousands of dollars plus whatever funds came from the EU and NWO.
I have to say I am confused about your reply in the first sentence you seem to acknowledge that the wordingwas related to the cost of "writing and testing" crypto software. However in the second sentence you seem to indicate that your thesis was about the switching costs users face. Which is it? You did not say I get to use nacl "more or less for free" you said that "people have been writing and testing small, auditable and usable open source crypto, more or less for 'free'." That quote seems to be about the cost of creation not the switching costs.
Do you think djb et al produced nacl "more or less for free?"
I mentioned "free" only to point out that there is no financial cost to switching to it. I guess I did not type the sentence with enough care; words are missing. My apologies.
I imagine people would be willing (and are accustomed) to paying for software of similar quality.
But I'm also wondering why this bothered you so much.
Does it make a difference that grants were received?
Is the funding not transparent enough?
The blog article on OpenSSL mentions payments for consulting and "features" to be added to OpenSSL.
Should I be concerned about what those features are, and who is paying for them? Are you concerned?
I'm just nterested in cleaner code than OpenSSL's. NaCl looks cleaner to me.
Maybe I'm wrong. But I'd rather be compiling programs that use libnacl or some other simpler alternative than ones that use libssl.
We all have to make decisions about what software we choose to use, even if we are not cryptographers.
I see nothing wrong with discussing alternatives to OpenSSL. This bug has been a real PITA.
> I mentioned "free" only to point out that there is no financial cost
> to switching to it. I guess I did not type the sentence with enough
> care; words are missing. My apologies.
It speaks highly of your character that you say this to the jerk on the
internet said you were full of shit.
> But I'm also wondering why this bothered you so much.
Because crypto is important. A lot of harmful attitudes/mindsets are
reinforced if people think NaCl was created in the authors spare time
and did not require funding:
- "Why should I donate to GnuPG/OpenSSL/Tor/Mozilla(NSS)? Those NaCl
devs wrote NaCl for free."
- "How hard could it be to implement a crypto library? Nacl was a side project. The Nacl devs 'have day jobs in
academia' and created nacl in their spare time. They did it for free, so they obviously didn't need to spend money on
testing environment, research material or hire/consult experts. On the other hand look at SelfiesMadeEa.sy they
raised serious cash and had to quit their jobs because they tackle hard problems."
- "Obama and the rest of gubmint are taxing me to death. Government should be pay for the military and maybe some
roads; not waste money on liberal academics in ivory towers, maplethorpe and those pinkos from NEA or some stupid
robot/telescope that cant do metric conversions."
- "OMG NSA is evil. USA does nothing but invade countries and privacy."
> Does it make a difference that grants were received?
No it does not make a (negative/harmful) difference that grants were received. I think it is a shining example of
modern civil society; you have the US, NL and the EU teaming up to fund strong crypto by top notch folks from a
number of countries. Governments should fund research, applied and basic, and they should be encouraged to fund
more of it.
Somewhat tangential: Knowledge of the grants also seeks to eliminate the
idiocy in the latter two examples above. People need to be reminded that
big government is not always an evil force, governments can be a force
for good. I do not know if you saw my other comment about tor funding
but tor had revenue of \$2+ million in 2012 and 60% came from US
government. I don't know where you are from but I bet you have met a
simple minded moron wearing a tea party costume or trendy European
threads that will not stop complaining about the evil Obama surveillance
administration. Blow their minds and ask them to wrap their heads around
the:
- $800k from DoD for "Basic and Applied Research and Development in
Areas Relating to the Navy Command, Control, Communications,
Computers, Intelligence, Surveillance, and Reconnaissance"
or
- $350k from State for "Programs to Support Democracy, Human Rights
and Labor" and "New America Foundation: International Programs to
Support Democracy, Human Rights"
> Is the funding not transparent enough?
If this is in regards to the lack of numbers from NWO or the EU I am
sure that I am at fault. (I also think one of djb's EU grant numbers
might have a digit transposed) I imagine that the dutch version of
nwo.nl is easier to use.
> The blog article on OpenSSL mentions payments for consulting and
> "features" to be added to OpenSSL.
> Should I be concerned about what those features are, and who is paying
> for them? Are you concerned?
I think we should be concerned that OSF is not doing a better job
highlighting sponsors and attracting new ones. It should be easier for
someone with check writing authority at big.corp.com to stumble across
the sponsors information and think to themselves "hey, we should drop
some petty cash on these folks. We use the product and I bet the
marketing folks would appreciate the bump in visibility for a fraction
of the cost of our latest failed social network branding efforts." If I
was OSF I would look at the \$2 million tor brought in and ask myself
"maybe we could do a better job of sponsor outreach? Tor is important to
these people that wrote checks and tor uses libssl-dev, I wonder if
there is an opportunity?"
They say they have 100 pending contracts... I believe their funding strategy could be fixed by hiring people to do the contract hours for a salary (those don't need to be OpenSSL core hackers, just good enough with the core of OpenSSL to help customers and provide viable work when a modification is requested). With the money you earn you also pay the core team members to just work at OpenSSL with the right priorities: this is where you improve security, do refactoring, and trow away #ifdefs.
It's only the wrong funding model if there is a superior alternative. It's a non-ideal funding model, but as is the main point of the article, they are and have been actively looking for alternative sources of funding without success. I hope your gripe is not directed at OpenSSL but at those who could be supporting it financially.
Reminds me on an article about how short-term, market-driven university research is killing innovation in abstract fields with no apparent financial ROI (e.g. mathematics, physics, chemistry and biology).
If research was market-driven in 1859 Riemann might not had time to play with zeta functions because at the time no one knew what to do with them. Same for Bayes and so many others. It's not that those man had a better environment around them than current researchers, it's mostly that we're getting results slower than we might as a human kind.
It's true that their funding model is imperfect, but then other funding models are imperfect in different ways, and their model should be perfectly workable: spend six months of the year working on contract jobs and the other six months making general improvements to the code.
The reason it's not working is that their rates are far too low. This is, for whatever reason, a classic mistake of technical people. If you're turning down work for lack of time, you need to increase your rates until this stops happening.
One approach, if you're coordinating development through the foundation, might be for the foundation to require some percentage of the funds go to foundational improvements.
Could someone involved with the OpenSSL Foundation and the OpenSSL project maybe pitch in with a quick description of how the project is managed?
* Who owns which subsystems?
* Is there a board of governors or a BDFL or something like that effectively overseeing the whole project?
* What is the process for screening commits from people new to the project?
This whole post seems to be tinged with a bit of defensiveness on behalf of the most active committers to the project. But it wasn't the active committers who introduced this most recent bug.
The tor project is a great example of how open source software (OSS) projects can work with sponsors. Trying to find more information on Qualys and PSW Groups sponsorship of openssl is a nightmare compared to tor project sponsors.[^1][^2] Without the tor project's emphasis on transparency and professionalism I doubt they could post numbers like this:
Since meeting the revenue milestones of $1,253,241 in 2009,
$1,574,119 in 2010 and $1,681,101 in 2011, Tor has reached new
heights in 2012 with over $2 million in revenue (unaudited).[^3]
My comment should not be read as a criticism of OpenSSL, it should be interpreted as cause for optimism. The tor project has demonstrated that OSS projects can get sponsored to solve complicated security problems that are difficult to explain to the general public.
It's well and fine that Stephen lives very cheaply, but all of this is an attempt to distract from the OpenSSL project's very real issues by wearing a cilice then bitching about it.
The fundamental facts are these: openssl contains a large quantity of code that, if I where to check into my company's repo, I would have at best a rough conversation with the cto and at worst I'd get fired. Plus a lack of good tests. These combine to create more than hypothetical problems; we've seen some severe security holes and there's almost certainly more to come.
The question that should be discussed is if openssl is, ala sendmail, unsuitable for purpose and, if so, what should it be replaced with.
OpenBSD has had two holes in a heck of a long time. By contrast OpenSSL has had a remote execute in 2010, and another 4 in 2002, and is regularly patching DOS's resulting from memory corruption that turns out not to be exploitable.
It is 453,000 or so lines, more than five times the size of xv6. It is ten times as big as PolarSSL. Documentation and internal structuring is wildly inconsistent. Features that make static analysis annoying are widely used. The API is far too low level.
Do you believe this is acceptable in a security library? Do believe that aspiring to the security of qmail or OpenSSH is a reasonable goal, even at the cost of features? Why should I use OpenSSL for TLS termination when formally validated alternatives exist?
> Why should I use OpenSSL for TLS termination when formally validated alternatives exist?
Oh please do share! (spoiler: alternatives which enable side channels because the pretty compiled-optimized-code (that is generated from source code that itself may feature immutable data etc. with no side effects) is ripe with leaks via cpu branching, caching, etc etc do not count.) This is not a spiteful rhetorical inquiry, by the way!
Those side-channel attacks are largely theoretical. OpenSSL and RSA in general are vulnerable to side channel attacks, because many of the fundamental operations are not constant-time. That can change eventually, but RSA is difficult to write in a constant-time manner. OpenSSL certainly has had its fair of lab-demonstrated side-channel attacks, but I don't think anyone has been able to demonstrate their use in virtualized hosting environments against arbitrary other tenants.
Side channel attacks once you've already got code running on the same operating system as the target are much easier. But if you can get arbitrary code to execute on the same machine running OpenSSL, you probably already have their keys.
So, I think the criticism to OpenSSL is valid. Why use it when it seems there are less bad alternatives? A lot of it comes down to network effects, inertia, and licensing. That's a much better answer than side-channel attack surface area, which all RSA implementations share :)
> Those side-channel attacks are largely theoretical.
Before you write secure software, you have to consider who your adversaries are, and what they are capable of doing. Side-channel attacks over the network are definitely practical [1] [2] [3]. If you're making even a basic SaaS product, you should assume your adversaries can carry out the above. You should assume that by now, even lowly script kiddies have side-channel exploits in his arsenal.
Nope: you can exploit the timing attacks from across a network link in AES. DJB did this long, long ago. Furthermore, I believe OpenSSL has constant-time RSA, but you can always use ECDSA for which constant time implementations in C exist. (I wrote one, but I still need to switch the hash function to SHA256 and add the unnecessary encoding steps to the output, so consider it a PoC).
Are you sure? Searching for "side channel OpenSSL" reveals a majority of the attacks are against ECDSA. Of course, searches aren't the best measure of vulnerability, it's just an indication of popularity.
PolarSSL has been validated with FRAMA-C by a company. Unfortunately you have to pay to see the results, and they use blinding for bignum arithmetic, not constant-time. (But they do have a decent curve implementation). http://trust-in-soft.com/news/
To remove side channels and preserve validation, you need to be slightly clever. You validate that a C implementation of one of the functions does the same thing as a function in the validated package, then swap them in the build output. This can be done for miTLS. Your validation of the C part is much more limited in scope than the whole thing.
> To remove side channels and preserve validation, you need to be slightly clever. You validate that a C implementation of one of the functions does the same thing as a function in the validated package, then swap them in the build output. This can be done for miTLS.
I am not defending OpenSSL but I am not sure your comparisons are very informative and quite frankly some of your sloccounts seem to wander far away from fact.
> OpenBSD has had two holes in a heck of a long time
Two remote holes in the default install. The default install is configured in such a way as to minimize the attack
surface area. Do you know what services are configured to respond to public internet traffic in the default install?
OpenSSL on the other hand essentially is always interacting with public internet traffic. Do you use OpenBSD on your
desktop and laptop?
> 453,000 or so lines, more than five times the size of xv6
Are you referring to Xv6, "the simple Unix-like teaching operating system" a project designed for education and not
commercial service offerings? How does this comparison of the size of xv6/OpenSSL inform the debate? The Linux
kernel has 12 million lines of code. What should I conclude from this number compared to OpenSSL? Do you think some
of PolarSSL's failures with frankencerts can be explained by how small the code base is? Please see the addendum for
more sloccount discussion.
> Do believe that aspiring to the security...even at the cost of features?
I don't mean to be difficult but I have no idea what you are asking here. Which features are you referring to?
Without some specificity of "cost of features" this seems to be more about showmanship and rhetorical style than
fostering meaningful debate. Do you run OpenBSD on your desktop/laptop?
> Why should I use OpenSSL for TLS termination when formally validated alternatives exist?
What are these alternatives? I did not realize there were multiple formally validated alternatives to OpenSSL.
Maybe we have different opinions about "alternatives" versus "implementations."
SLOCCOUNT Addendum:
I did not want to inject a pile of sloccount data in the main body of my comment but I do have some questions about
your counts. After a little research it seems that your numbers might be somewhat hand wavy which is unfortunate
because you gave no indication of this in your comment.
What versions of OpenSSL and PolarSSL are you referring to? My count for OpenSSL 1.0.1g puts it at 6.5 times the
size of PolarSSL and not the 10 times that you stated in your comment. I could not connect to OpenSSL.org so I do
not know if they have released a version since April 7th but I must say I am surprised that that the OpenSSL dev
team added 91,344 lines (25% increase) in the last seven days. And even if they did that still puts them 90,000
lines short of "ten times the size of PolarSSL."
My sloccount data:
# OpenSSL 1.0.1g:
Totals grouped by language (dominant language first):
ansic: 273820 (75.71%)
perl: 69192 (19.13%)
asm: 11015 (3.05%)
cpp: 4367 (1.21%)
sh: 3238 (0.90%)
lisp: 24 (0.01%)
Total Physical Source Lines of Code (SLOC) = 361,656
# PolarSSL 1.3.6-gpl
Totals grouped by language (dominant language first):
ansic: 52212 (94.71%)
sh: 2165 (3.93%)
perl: 748 (1.36%)
tcl: 4 (0.01%)
Total Physical Source Lines of Code (SLOC) = 55,129
# Linux kernel 3.14
Totals grouped by language (dominant language first):
ansic: 11828087 (97.00%)
asm: 275047 (2.26%)
!!!TRIMMED counts < 0.5%!!!
Total Physical Source Lines of Code (SLOC) = 12,193,312
# Xv6
Totals grouped by language (dominant language first):
ansic: 7022 (87.81%)
sh: 481 (6.01%)
asm: 301 (3.76%)
perl: 189 (2.36%)
lisp: 4 (0.05%)
Total Physical Source Lines of Code (SLOC) = 7,997
You are correct: I meant 10x the size of xv6 and 5 times that of PolarSSL. I used wc, so comments and blanks count. xv6 implements a multiuser operating system. This is not an easy undertaking by any means, and far more complex than TLS in terms of conditions to satisfy.
As for the rest of your comment, I have used OpenBSD on the desktop. I have never found the lack of a webserver running, or anything but sshd to be a hindrance to making a machine on which with a browser I can read papers, bank, listen to music, and program. sshd is of course attackable by anyone: good luck getting in though. Other OS's have followed OpenBSD, as do most hardening guides.
One of the features OpenSSL has is support for ciphers that long ago have been disabled, together with out-of-date TLS versions such as SSLv2(!). TLS is a specification, with multiple implementations. The specification is of poor quality, the protocol worse, but OpenSSL is in a class by itself, including implementations of national ciphers no one uses.
While Frankencerts were an issue, there was no realistic way to turn the two incorrect "OKAY" and the one "NOT OKAY" result of PolarSSL into an attack. You would need a CA to issue a cert with a start time in the future. It also does not take an enormous amount of code to fix this issue.
This is the wrong funding model. It keeps money in OpenSSL developer's pockets, but there is no financial incentive for any OpenSSL developer to work on foundational improvements to OpenSSL. He said himself: there is over $100,000 in open contracts for competent developers to work on non-foundational improvements to the project. If you are an enterprising developer with good C skills and a knack for crypto projects and you apply to work for the OpenSSL foundation, are you going to start servicing that $100,000 pool of contracts or are you going to pretend that money doesn't exist and live on ramen?
If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to meet their particular needs, then none of that money is going to developers to strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing. It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else.
Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?
No one will pay an hourly rate to accomplish these tasks. Google is not going to pay by the hour for a developer to stare at a function until they grok it; they want a feature. Joe Company will not pay for developers to write unit tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know how to make their code handle it.
This model needs to go away. Competent OpenSSL developers time is too valuable to waste on client asks. Their project is too important, and as long as the money is flowing only for novel features and not structural improvement, then that money will dictate that only new features are developed.