tag:blogger.com,1999:blog-8331743177423628742024-03-18T23:18:25.040-07:00Insane Coding'coz good thinking requires going outside the boxinsane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.comBlogger77125tag:blogger.com,1999:blog-833174317742362874.post-36008887477449783182019-02-17T09:02:00.000-08:002019-02-17T13:20:01.584-08:00Is SHA-3 (Keccak) already broken?<style type="text/css">p { margin-bottom: 0.1in; line-height: 115%; background: transparent none repeat scroll 0% 0%; }</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
The other day I needed to add <a href="https://en.wikipedia.org/wiki/SHA-3">SHA-3</a> (Keccak) support to a project in
order to ensure interoperability with some other applications. I
actually wanted to add SHA-3 support back when it first came out
before I had a specific use for it, but at the time, I looked at some
code samples, and it looked way too confusing. However, when looking
for code samples the other day, I found some more straightforward
implementations, that were clear enough that I could figure out how
it worked and write my own implementation.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
I’ve implemented
over a dozen hash algorithms, for reasons I won’t go into here. One
upside to doing this is that I can get a decent understanding of how
it works internally so I can be familiar with its pros and cons. Now
understanding how SHA-3 (Keccak) works, and comparing it to some
other hash algorithms, I’m actually somewhat appalled by what I’m
seeing.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<h3 style="line-height: 100%; margin-bottom: 0in;">
Hash Properties
Overview</h3>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<a href="https://en.wikipedia.org/wiki/Cryptographic_hash_function">Cryptographic hash algorithms</a> (like the <a href="https://en.wikipedia.org/wiki/Secure_Hash_Algorithms">Secure Hash Algorithm family</a>) take a file,
document, or some other set of data, and produce a fixed size series
of bytes to describe the data. Typically between 20 and 64 bytes.
These hashes are supposed to change drastically if even a single bit
is altered in the input data that is being hashed. They’re also
supposed to ensure it’s computationally difficult to produce data
targeting a specific hash.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
There’s <a href="https://en.wikipedia.org/wiki/Preimage_attack">two main properties</a> desired in a cryptographic hash algorithm:</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
1) That it should be
too difficult to generate two documents that have the same hash. This
ensures that if party A shows party B a document, and gets them to
approve the document’s hash in some sort of <a href="https://en.wikipedia.org/wiki/Whitelisting">whitelist</a> or <a href="https://en.wikipedia.org/wiki/Digital_signature">signature</a>,
that party A cannot use a different document against B’s whitelist
or signature that is now also approved without consent of B.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
2) That if a certain
set of data is whitelisted or signed by a party, that it would be too difficult to generate a different set of data that
matches a whitelisted or signed hash.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
These two properties
are really similar. The only difference is whether you’re targeting
a preexisting hash that exists somewhere, or generating two different
sets of data that can have any hash, as long as they’re identical.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
If the first
property is not applicable for a hash algorithm, it may still be
usable for cases where you’re not accepting data generated by a
third party, but only validating your own. If the second property is
not applicable, there may still be some use cases where the hash
algorithm can be used, especially if only a certain subset of hashes
can have data generated against them which match. However, if either
property does not hold true, you generally should be using some other
hash algorithm, and not rely on something which is broken.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<h3 style="line-height: 100%; margin-bottom: 0in;">
Classical
Cryptographic Hash Algorithm Design</h3>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
For decades hash
algorithms basically did the following:</div>
<ol>
<li>Initialize a
“state” of a particular size, generally the output size, with a
series of fixed values.</li>
<li>Break the data up
into a series of blocks of a larger particular size.</li>
<li>Take each block
and use bit operations and basic math operations (addition,
subtraction, multiplication) on its data, ignoring <a href="https://en.wikipedia.org/wiki/Integer_overflow">overflow</a>, to
reduce the block to a much smaller size, generally the output size,
to be merged with the “state”. Each block is processed with some
fixed data.</li>
<li>Combine each of
those states. This might be done by <a href="https://en.wikipedia.org/wiki/Exclusive_or">xoring</a> or adding them altogether,
and throwing away any overflow.</li>
<li>Append the size
of the data to the final block, producing an extra block if neccesary, performing steps 3 and 4 upon it.</li>
<li>Return the state
as the result.</li>
</ol>
<h3 style="line-height: 100%; margin-bottom: 0in;">
</h3>
<h3>
</h3>
<h3 style="line-height: 100%; margin-bottom: 0in;">
SHA-3 Competition</h3>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
All hash algorithms
sooner or later don’t quite hold up to their expectations. Although
this isn’t always a problem. For example, if a hash algorithm is
supposed to take 1000 years to <a href="https://en.wikipedia.org/wiki/Brute-force_attack">brute force</a> a match to an existing
hash, and someone figures out an algorithm to do it in 950 years, it
doesn’t provide the security it theoretically advertises, but the
margin of security is so high, it doesn’t matter.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
However, in some
recent years, real attacks violating one of the two key cryptographic
hash properties which could be performed in hours or even minutes
have been found against the popular <a href="https://en.wikipedia.org/wiki/MD5">MD5</a> and <a href="https://en.wikipedia.org/wiki/SHA-1">SHA-1</a> algorithms. These
attacks don’t necessarily invalidate MD5 and SHA-1 from every
potential use, but it’s bad enough that they should be avoided
whenever possible, and should not be relied upon for security.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
There’s also an
issue with these classical hash algorithms regarding how the states are chained and returned as the hash. It
makes it easy for hashes to be misused in some cases. Someone who
doesn’t necessarily know what data matches a particular hash, can
still <a href="https://en.wikipedia.org/wiki/Length_extension_attack">calculate the hash of data + data2</a>. This might be an issue in
certain naïve usages of these hash algorithms.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Further, all the
classical algorithms were shown to not quite hold up to their
expectations, even though they’re still considered secure enough
for the time being. This led to a desire to create a new series of
hash algorithms which have a different structure than the classical
ones. Therefore a <a href="https://en.wikipedia.org/wiki/NIST_hash_function_competition">competition was held</a> to create some new ones and determine the best candidates for future use and to earn the title “SHA-3”.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<h4 style="line-height: 100%; margin-bottom: 0in;">
BLAKE and BLAKE2</h4>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
One of the SHA-3
competition finalists was <a href="https://en.wikipedia.org/wiki/BLAKE_(hash_function)">BLAKE</a>. This hash algorithm was composed of
a few different components. Its authors suggested studying
BLAKE leaving out each of them, to understand how strong BLAKE was based on a
majority of its components. After the competition was over, and after much
study, it was realized that one of the components of BLAKE wasn’t
necessary, and that the amount of operations overall could be reduced
to improve its speed, without really hurting its security. Taking that alongside some interesting ideas
introduced alongside <a href="https://en.wikipedia.org/wiki/Skein_(hash_function)">Skein</a> and Keccak, two other finalists, BLAKE2
was created. BLAKE2 is one of the fastest cryptographic hash
algorithms, and is also considered to be one of the strongest
available.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
BLAKE2 works as
follows:</div>
<ol>
<li>Initialize a
“state” of a particular size, with a series of fixed values,
mostly zeros.</li>
<li>Break the data up
into a series of blocks of a larger particular size.</li>
<li>Take each block,
the block number, and a flag, and use bit operations and addition,
ignoring overflow, reducing the size of these in half, to be merged
with the “state”. Each block+number+flag is processed with some
fixed data.</li>
<li>Combine each of
those states.</li>
<li>The flag used alongside the final block is different than all prior
blocks.</li>
<li>Return the state
as the result.</li>
</ol>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Theoretically BLAKE2
is stronger than classical hashes, because there is data added to
each block that is processed, which cannot be simply influenced by
passing data to it. This makes computing data to go from state A to
state B more difficult, because you would need a different set of
data to do so depending on where the block you’re trying to replace
is. Calculating a hash from another hash for data + data2 is more
difficult, because of the flag change. The state for data would be
different if more blocks were appended to the data.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
</div>
<h4 style="line-height: 100%; margin-bottom: 0in;">
Keccak</h4>
<h4>
</h4>
<div style="line-height: 100%; margin-bottom: 0in;">
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
The actual winner of
the SHA-3 competition was the Keccak algorithm. It was chosen because
it was really different from classical hashes (not necessarily a good
thing), and really fast in hardware implementations.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Keccak works as
follows:</div>
<ol>
<li>Initialize a
large “state” of a particular size, with zeros.</li>
<li>Break the data up
into a series of blocks of a smaller particular size.</li>
<li>Take each block,
and use bit operations, to be merged with the larger “state”.
Each block is processed with some fixed sparse data.</li>
<li>Combine each of
those states.</li>
<li>The final block
has two bits flipped.</li>
<li>Return a small
portion of the state as the result.</li>
</ol>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Like BLAKE2, Keccak
aims to be stronger than classical hashes because there is state data
larger than the size of the block, which cannot be immediately
influenced by data (but can still be influenced). Calculating a hash based upon another with
appended data is also difficult, because the result is a truncated
state. The bit flips in the final block would help as well.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<h3 style="line-height: 100%; margin-bottom: 0in;">
My thoughts</h3>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
After implementing
Keccak and understanding its design, I became alarmed by how much is
missing. The use of pure bit operations make it easier to compute in
reverse, aside from a few <a href="https://en.wikipedia.org/wiki/Bitwise_operation#AND">AND operations</a>. The computation of the
final state uses bit flippings inside the block, as opposed to
outside beyond it, making it easier to tamper with (although
theoretically still difficult). But most importantly, the utter lack
of using a block counter or data size anywhere.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Classical hash
algorithms make it difficult to insert blocks anywhere in the
middle of data and get a matching hash. Even if you were able to
compute some data to go from state A back to state A, you could not
insert this somewhere, because the data size at the end must then be
different, resulting in a different hash. Same goes for BLAKE2
because of its block counter appended to each block it processes.
<b>Keccak has absolutely nothing here.</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Keccak’s initial
state is all zeros, something which every Keccak hash must use. If
you could compute data which is a multiple of the block size which
when processed in Keccak would go from state all zeros to state all
zeros, <u>you have just broken Keccak</u>. You would be able to prepend this
data as many times as you want to the beginning of any other set of
data, and produce the exact same hash. <b>This would apply to every
single Keccak hash produced, it targets all Keccak hashes in
existence.</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Now, I’m no expert
cryptographer. Perhaps Keccak’s bit operations ensure that a state
of all zeros could never be produced from it. Maybe there even exists
a proof for this which is in a document or article I haven’t seen yet.
However, if there is no such proof, <u>then Keccak may be broken much
worse than even the classical hash algorithms are</u>. With the classical
algorithms that are broken, you generally have to break each hash
with a very large amount of computations specific to each hash result
you want broken. With Keccak here, once a prefix becomes known, you
can just prepend it, <b>no computation necessary</b>.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<h4 style="line-height: 100%; margin-bottom: 0in;">
Improving Keccak</h4>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Of course if Keccak
is indeed broken in this fashion, it’s not hard to fix. The data
size could be processed in the end, or a block counter could be used
alongside each block like BLAKE2, the state is certainly large enough
to be able to handle it. If one were to change Keccak, I’d also
move the bit flippings at the end to occur beyond the block size
inside the state, just to make it more difficult to tamper with the
final result. In the mean time though, <u>is Keccak, and therefore
SHA-3, even safe to use?</u></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<h3>
Summary </h3>
For the three kinds of length extension attacks
that may exist (prepending, inserting, appending), it appears that
classical hashes defeat the first two and have some issues with the
third. BLAKE2 defeats them all. Keccak does an okay job against the
third, but offers no protection against the first two. A known attack
existing against the first would be catastrophic. This is fixable if the algorithm is improved.</div>
insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com11tag:blogger.com,1999:blog-833174317742362874.post-31324105378813207812016-04-24T18:26:00.000-07:002016-04-24T19:34:44.515-07:00Suggestions not to make when starting a new programming jobWhen starting a new programming job, it can be risky to suggest certain things. Others at the company do not know you well enough, and saying the wrong thing will immediately create a bad first impression, branding you as too inexperienced for your role or not a team player. Make one of the following suggestions, and your new job may not even last you a week.<br />
<br />
<h4>
Let's rewrite the codebase for the application in another <a href="https://en.wikipedia.org/wiki/Programming_language">language</a>.</h4>
If you weren't hired as a design consultant or to re-engineer the big-picture, suggesting to rewrite a key application in another language is almost certainly going to get you fired close to immediately.<br />
<br />
There can be many good reasons why an existing code base is in a particular language:<br />
<ul>
<li>It's a language all the other developers at the company already know.</li>
<li>It's the only language certain key libraries exist in.</li>
<li>The language for whatever reason excels at what is being accomplished with it.</li>
</ul>
In any of these situations, your suggestion is saying to switch to something the existing talent is unfamiliar with, or an admission of not understanding the scope of what is being done. In the former, you've just admitted to not being a team player, in the latter, you've admitted to having too little experience. Further, such a suggestion will indicate you probably are not as experienced with the language as your resume or initial interview seemed to convey, and perhaps you're a liar as well.<br />
<br />
Additionally, even if none of the above reasons are valid for the case at hand, no manager wants to delay a project to rewrite everything. Furthermore, rewrites are nearly every manager's nightmare, as they fear new bugs being introduced. All this combined, expect little job security when making such a suggestion early on.<br />
<br />
If you're tasked from the get go with fixing some small script, you can probably get away with changing the language - if you get approval first. But ask about it in a way where it is clear you are concerned what is best for maintaining it by others in the future, not as what is easiest for you and shows indifference to the company's needs.<br />
<br />
This applies to any sizable rewrite in general. Few will care about rewriting some small script or component. But offering up front when no one knows you to rewrite a large project, even in the same language, is just asking for trouble. Sizable rewrites normally aren't done unless there's a team in place that are already known to be able to work together, and can do so reasonably quickly and with relatively few bugs in the process.<br />
<br />
<h4>
Let's switch to another <a href="https://en.wikipedia.org/wiki/Version_control">version control</a> system.</h4>
You may find yourself on your first day having to use a version control system you are unfamiliar with. Do not blurt out that you'd prefer an alternate system. Doing so is tantamount to demanding the rest of the company cater to your familiarity or preferences, instead of you trying to conform with the rest of the company.<br />
<br />
Existing version control systems may also be entrenched into much of the existing infrastructure and work-flows. There can be many scripts in place that were written long ago and only work with the particular version control system in use. The entire business may even depend on these scripts for managing all deployments or builds. If so, you just asked to flush the entire company down the drain.<br />
<br />
I've had the recent displeasure of hiring a short-lived employee who demanded and fought for us to switch to Github from his second day at work, before even learning about what our needs were and if Github was or wasn't a good fit. Github may be terrific for working with <a href="https://en.wikipedia.org/wiki/Git_%28software%29">Git</a> for an open source project, but it's not necessarily a good fit for proprietary projects. Github for private repositories costs money, and even if it were free, you don't necessarily want to share your code with a third party when you are quite capable of locally hosting Git repositories and tools. It was pretty clear from this employee's passionate backing of Github that he had little idea of what our source code hosting objectives were, too inexperienced to realize there are considerations other than the features Github offers, and his passion revealed he wasn't going to be a team player. He was let go fairly quickly.<br />
<br />
<h4>
Let's move everything to another hosting provider such as <a href="https://aws.amazon.com/">Amazon Web Services</a>. </h4>
Amazon's web services provide a continuously growing offering of all kinds of web services to run your business. It is getting quite popular due to being a <a href="https://en.wikipedia.org/wiki/One_stop_shop">one-stop-shop</a> for many kinds of businesses, and offering very low prices for various use-cases. Amazon also provides elaborate APIs for developers to make use of in order to manage their services and maintain interesting software. Similar kinds of hosting services are showing up from Microsoft, Google, and others as well.<br />
<br />
Some developers who have experience with these services want everything hosted there due to their existing experience or ease they know they can get certain tasks accomplished with them. But that doesn't mean these services are necessarily right for a business.<br />
<br />
These services, such as AWS, bill according to every kind of individual usage they can meter. Their costs are very low and highly competitive for minimal usage. However, once you get into large amounts of usage, competing services from smaller companies which offer flat rates may be more cost effective. Some business will also choose a flat rate service over AWS and those like them, because with those metered services, pricing is unpredictable. One doesn't necessarily know up front how much usage will be occurring, and that can make it difficult to work out a budget or be profitable with certain pricing models for applications built upon these services. These businesses will choose a flat rate model even if AWS or others like them are cheaper, simply because there are no surprises later.<br />
<br />
Whatever existing service is in use may also be entrenched for a variety of reasons. So before making a suggestion which may show you have zero business sense, or disregard for how things are currently running, get an understanding of what is presently being used and what the pros and cons of a switch are. There can be very good reasons why some hosting company's service is not already in use by the company before you arrived at your new job.<br />
<br />
<h4>
Let's switch the servers to a different Operating System, Application Server, or Database System.</h4>
For many of the same reasons described by previous statements, such a suggestion is revealing inexperience, incompetence, or a demand for everyone else to revolve around you. Various OSs, servers, or DB systems are better geared towards working in certain areas. Demanding a change will make people think you don't understand why something is being used. Even if you are correct that something else is better, the existing system is probably entrenched, or the existing talent can't work with anything else. Such demands only show you have little business sense, or can't be a team player. Unless you're specifically asked for an opinion in these matters, don't offer one!<br />
<br />
<h4>
Conclusion</h4>
Nobody likes change. Change is hard. Change may even be a very bad idea. Change can fail. Others may be unable to adapt to change.<br />
<br />
Before suggesting changes like those above or similar things, really understand why something is being used, and what others are able to deal with. If you can't work out the pros and cons or understand what the impact of a change is going to be, nobody wants your opinion. If you're a new and lowly employee on the corporate ladder, it is best to keep your mouth shut.<br />
<br />
If you feel strongly a change is needed, this is usually an indication you are inexperienced or cannot be a team player. Experienced programmers will find a way to do almost anything with anything, they aren't locked into a single tool to solve a problem. If you've been somewhere a while, and your colleagues respect you, then approach these things very carefully. Ensure your proposal is in the spirit of being a team player and in the company's best interest. Otherwise, prepare for a <a href="https://en.wikipedia.org/wiki/Pink_slip_%28employment%29">pink slip</a>.<br />
<br />
If you're a manager, it may be useful to note that all these above kinds of statements, <b>if made early on</b>, are symptomatic of programmers who are incompetent. Probably best to fire them immediately and not waste resources on them that can be better spent training their replacement.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com222tag:blogger.com,1999:blog-833174317742362874.post-77053623638856492582016-04-09T20:28:00.000-07:002016-04-24T13:15:09.714-07:00OAuth - Why it doesn't work, and how to Zero-day attack existing services<div style="text-align: left;">
Since this article is extremely long, it is also available in a <a href="http://no-oauth.insanecoding.org/OAuth%20-%20Why%20it%20doesn%27t%20work,%20and%20how%20to%20Zero-day%20attack%20existing%20services.pdf">PDF version</a>. This article also has a <a href="http://no-oauth.insanecoding.org/">companion website</a>.</div>
<hr />
<h2 style="text-align: center;">
</h2>
<h2 style="text-align: center;">
Intro</h2>
<h3 style="text-align: center;">
What is OAuth?</h3>
Let me begin with a quick run down on what we're discussing. <a href="http://en.wikipedia.org/wiki/OAuth">OAuth</a> is a family of proposals that is being used for designing different kinds of authorization and authentication systems, and framing the usage for <a href="https://en.wikipedia.org/wiki/Application_programming_interface">APIs</a> and how different systems are integrated.<br />
<br />
OAuth exists in many flavors, version 1, 1a, 2, and other specifications based on top of parts of it. Its exact usage also greatly differs across the many different implementations out there. <br />
<br />
<h3 style="text-align: center;">
Disclaimer</h3>
<a href="http://insanecoding.blogspot.com/2013/03/oauth-great-way-to-cripple-your-api.html">I've written about OAuth in the past</a>, and I've gotten a lot of feedback from readers all over. To avoid some criticism my past article received, I'd like to point out that this article will be focusing on how OAuth is typically used, and most of the points discussed here are true of nearly every major service which leverages OAuth in some way.<br />
<br />
To put this differently, not every platform utilizing OAuth is necessarily broken. Due to the various flavors of OAuth, and the <a href="https://tools.ietf.org/html/rfc6749">76-page document on different possibilities with OAuth 2.0</a>, it is possible to create something secure and sane to use which conforms to something OAuth based. Therefore your favorite flavor and design with OAuth may escape some or all of the problems discussed herein, even though the odds are that it does not.<br />
<br />
Some of you may also argue that some things done with OAuth are misusing the specifications, or that OAuth doesn't specifically mandate things be done the way they are. Whatever the case may be, I'm not here to write about a particular OAuth based specification, but how OAuth is currently utilized by major players in the market, regardless if what these organizations are doing conforms with the specifications or not. This will impact many readers, either because they use services making use of OAuth in these ways, or they're maintaining OAuth-based services, and are building platforms similarly to how many others are building them.<br />
<br />
<h3 style="text-align: center;">
Article structure</h3>
This article is going to be a long one, and in fact most sections in this article cover topics with enough material to make an entire article themselves, so let me give a brief outline on what this article is about and how it will be organized.<br />
<br />
This article is the presentation of the culmination of research into what is currently being done in the market with OAuth. Not everything in this article is consistent with every product making use of OAuth, but this article will present what was found to be common practice and the biggest culprits behind insecure or useless services.<br />
<br />
After this introduction, I'm going to begin by analyzing security flaws with OAuth. The general principles behind some of these flaws are well known by those in the security community, and already briefly analyzed in existing publications. However, I'm going to be covering some cases that have not yet been publicized, and also elaborating upon some points of well known flaws to make them easier to understand to the average developer or manager and emphasize why they need to be concerned.<br />
<br />
After that I'm going to present an analysis of how some key components of OAuth are popularly implemented and how the popular implementations cripple services which choose to use OAuth by severely, inappropriately, and undesirably limiting what can be accomplished with them. Some techniques will be discussed that can be used as the basis for workarounds in a limited amount of cases, with a focus on the absurdity involved in implementing such. As part of the above, it will be frequently pointed out how those using OAuth are hurting themselves and their business.<br />
<br />
Lastly I will briefly cover a few cases for when and how OAuth can be used properly, along with viable alternatives to OAuth which are currently in use. I will provide a survey of alternative techniques that will include what serious companies like Amazon and others are doing to achieve secure, usable, and reliable APIs.<br />
<br />
<h3 style="text-align: center;">
<a href="https://en.wikipedia.org/wiki/Responsible_disclosure">Responsible Disclosure</a></h3>
Several of the flaws with OAuth <i>as is it popularly used</i> today can be exploited to carry out strong attacks against major services. Attacks against OAuth are nothing new, as IBM, Oracle, and other concerned members of the IETF issued a <a href="https://tools.ietf.org/html/rfc6819">71-page document describing 50 classes of attacks against OAuth</a> based services over three years ago, and I already discussed some of these points in my previous article. Despite this though, major security flaws with OAuth-based systems are extremely common.<br />
<br />
I've spoken to executives and developers at several major corporations to point out security flaws with their OAuth-based systems (in one case, 4 years ago), and not one of them has done anything to fix their systems. It seems as though, given the popularity of OAuth, they haven’t heard of any of the alternative <i>viable</i> solutions, and they therefore assume that OAuth must be the most secure thing available. It also seems as if they either have not heard or shrug off the information documenting the demonstrated attacks against the core principals of OAuth. I'm hoping that wider public dissemination of this information will give those affected the proper kick in the pants that they need, and serve as a wake up call to those designing or maintaining services.<br />
<br />
Therefore, if you investigate and find that some or all of these flaws exist in your service which either provides or makes use of something OAuth-based, please act responsibly in dealing with this information. Update your services appropriately, or apply appropriate pressure to your business partners to address the relevant issues.<br />
<br />
Although the various information mentioned here and linked to from here can be used to exploit existing services <i>today</i>, please be responsible and aim to improve and not destroy what belongs to others. This article is meant as a wake up call for those implementing their services improperly and an appeal to improve them, not a how-to guide for hackers looking to exploit them.<br />
<br />
<h3 style="text-align: center;">
</h3>
<h3 style="text-align: center;">
Usage Scenarios</h3>
For this article, I'm going to focus on two usage scenarios and see how OAuth stacks up with them and why it doesn't work. It is important to keep these scenarios in mind, as I will be constantly returning to them throughout this article.<br />
<br />
Let us imagine an Exciting Video Service (or <i>EVS</i>), where users can upload videos and share those videos with their friends, providing either public or restricted access to their uploads. EVS also provides OAuth-based APIs for uploading and deleting videos, and managing permissions regarding who can view those videos.<br />
<br />
While I will be focusing on this imaginary service for sake of example, the issues under discussion will apply to any service, whether it's for file and document storage, calendar management, online meetings, discussion groups, resource management, or anything else that provides an OAuth-based API. Also bear in mind that I'm not actually referring to any specific video service, even though some or all of these issues may apply to existing video services which make use of OAuth. It can be left as an exercise for the reader to determine which ones.<br />
<br />
For our first usage scenario, let us imagine a <a href="https://en.wikipedia.org/wiki/Camcorder">camcorder</a> manufacturer that would like to supply software with their camcorder which, among other things, allows videos recorded with the device to be uploaded to EVS. It is intended to allow users of this camcorder to plug their device into their computer, open the custom software, select the videos on the device they'd like to upload, and come back later knowing that all the selected videos will have been uploaded to EVS.<br />
<br />
For our second scenario, let us imagine a small organization decides to purchase 50 accounts with EVS for their employees, so all employees can upload videos and have them shared with other employees from the same department. This organization is using A Friendly Custom Platform, for managing their employees and which sub-units they belong to, and would like to integrate their AFCP service with EVS. This organization will expect that when a manager assigns someone to the sales department using AFCP that the employee in question will automatically gain access to all videos that belong to members of the sales department. They will expect the reverse to occur if they remove someone from the sales department, and all similar scenarios.<br />
<br />
<h2 style="text-align: center;">
The Issues</h2>
<h3 style="text-align: center;">
Security Related</h3>
<h4>
Stealing credentials / Gaining elevated access</h4>
One of the most popular reasons why <a href="https://en.wikipedia.org/wiki/Security_token">token-based authentication</a> systems (a core premise of OAuth) are currently used is that <i>when implemented properly</i> they avoid the need to provide third party applications and services with individual users’ credentials. It is undesirable to provide third parties with personal user credentials, because:<br />
<ul>
<li>It gives third parties more access than they require.</li>
<li>It is another place personal credentials are stored and can be stolen from.</li>
<li>It requires that API consumers be updated when users change their credentials.</li>
<li>Access cannot be easily revoked from just one application without also revoking access from all other applications.</li>
<li>User credentials can be too limited where additional authentication factors are in use.</li>
</ul>
The above list of problems can be avoided by any token-based authentication system, not just OAuth. While this is counted as a strength to OAuth, it is hardly unique, and viable alternatives can be used which carry the same strengths without OAuth’s weaknesses. <br />
<br />
However, despite being based on a solid premise, OAuth <i>as popularly implemented</i> attempts to avoid the above problems by providing a system with something along the lines of the following steps:<br />
<ol>
<li>Users visit the third party application/service such as AFCP and informs it of their desire to integrate a particular service.</li>
<li>AFCP then brings up a special login page hosted by EVS where the user enters their EVS credentials.</li>
<li>EVS then asks the user to confirm they are sure they want the third party application/service, AFCP, to have all the levels of access specified.</li>
<li>EVS then provides AFCP with some kind of token or series of tokens it can then use to make the various API calls.</li>
</ol>
Since these tokens are not the user's credentials, and can be specific to every user and application combination, limited in their permissions, and later revoked, it seemingly avoids all the aforementioned problems this setup was designed to address. However, in reality, despite having a sound premise at its core, this particular usage flow used by popular implementations of OAuth does not address all the above enumerated issues.<br />
<br />
This design begins from an insecure standpoint. The rule of thumb when it comes to designing a secure platform is that anything which begins from an insecure standpoint is already lost, it cannot be salvaged. Thanks to step #1, which begins on a service making use of EVS rather than EVS itself, users have already been <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">man-in-the-middled</a> from the very beginning. This is the computer-system equivalent of giving out personal or financial information over the phone to an incoming caller who claims to be from some utility service you use but whose number you either do not recognize or is blocked. There have been many such scams in recent times, and don’t need to be elaborated upon here. The main point is that if you cannot trust the party which initiated a connection, then you cannot trust the connection at all. It is impossible to design a secure authentication system for API use which achieves the objectives enumerated above unless the first step begins from the EVS side itself.<br />
<br />
Similar to the phone scam example, where the caller simply needs to sound as if they are calling from the company they’re impersonating, to attack a user, a third-party application or service just has to provide something which looks like the special EVS login page for step #2. Most users will not be aware that the page displaying the EVS logo and asking for their username and password is not authentically from EVS. Once a third-party service gains the user’s credentials in this manner, the application or service, and those behind it, now have more access than they should, and their access cannot be revoked without the user changing their credentials.<br />
<br />
Since users have already been conditioned with OAuth-based login flows with all kinds of embedded frames with corporate logos, silly pop-ups, and redirects with ridiculous URLs on atypical domains, most users won’t even notice any red flags which would alert them that the page into which they’re currently entering their credentials may not be genuine. A colleague of mine put it so: <u><i>The companies themselves are training their users to be </i></u><u><i><u><i><a href="https://en.wikipedia.org/wiki/Phishing">phishing</a></i></u> targets</i></u>. If URLs are needed to be displayed, the attacker can even register an official sounding domain (microsoft-authentication.us, ibm-secure.co.uk, gooogleauth.us, my-citrix.au, globalauth.world) and can even have redirection go through some URL which appears to be legit. <br />
<br />
In the case of actual standalone applications, such as that provided with your camcorder, nothing outrageous even needs to be done. Since the web browser is being provided by a custom application, it can already capture every single input box and all data sent over the network with it, it doesn't even need to <a href="https://en.wikipedia.org/wiki/Website_spoofing">spoof</a> some login form.<br />
<br />
This class of attack is labeled in the aforementioned security document as <a href="https://tools.ietf.org/html/rfc6819#section-4.1.4">4.1.4. Threat: End-User Credentials Phished Using Compromised or Embedded Browser</a>. The solutions offered? (emphasis mine)<br />
<blockquote class="tr_bq">
<b>Client applications should avoid directly asking users for their credentials</b>. In addition, end users could be educated about phishing attacks and best practices, such as only accessing trusted clients, as <b><u>OAuth does not provide any protection</u> against malicious applications</b> and the end user is solely responsible for the trustworthiness of any native application installed.
</blockquote>
Also:<br />
<blockquote class="tr_bq">
Client <b>developers should not write client applications that collect authentication information</b> directly from users and should instead delegate this task to a trusted system component, e.g., the system browser.</blockquote>
Essentially OAuth security guidelines say that developers making use of OAuth should not try to attack the users or do anything malicious. Relying on external developers not to do anything malicious is not a security model any sane service designer would rely on.<br />
<br />
Nearly every major OAuth-based service I know of can be attacked with the method outlined here.<br />
<br />
For those of you thinking OAuth is the new gold-standard for security, wake up! OAuth <i>as popularly implemented</i> is already defeated before it has begun. Many systems implemented way before OAuth ever existed are secure and work around this issue effectively. Unfortunately, and all too often, I’ve seen services transition themselves from something secure to an insecure OAuth model because someone told their developers or managers that OAuth was “<i>more secure</i>”, “<i>forward thinking</i>”, “<i>future proof</i>”, or any number of other buzzwords which sound nice but lack anything in the way of substance. Most of the time these changes are implemented without even reviewing whether these changes address any existing problems or if the solution is any better than what they are replacing.<br />
<br />
<h4>
Masquerading as an OAuth-using service</h4>
A common mistake I see in OAuth-based service design is the supplying of an endpoint designed for a web browser which accepts as one of its parameters a <i>client_secret</i> (or something of the same concept). The OAuth <i>client_id</i> and <i>client_secret</i> parameters are essentially a third-party platform’s equivalent of its own personal API username and password, and should therefore only be known to those developers making use of EVS's APIs. Since it is analogous to a password, the <i>client_secret</i> parameter should <b>never</b> be sent across a user’s web browser (<i>hint: the word <u>secret</u> is in the name of the parameter</i>). If some user of an application or service can find out the <i>client_id</i> and <i>client_secret</i> of the application or service, that means they can masquerade as that service and potentially do something malicious. Also note that some services will sometimes name the <i>client_secret</i> parameter something else, so review the service you are working with carefully and see if any of their other parameters need to be kept secret. Unfortunately, since important variables are sometimes not indicative of their nature, this problem is more common than it should be. Additionally, some services will build an authentication flow on top of OAuth using the <i>client_id</i> alone. Be wary of these, because under certain circumstances such a <i>client_id</i> functions exactly like a <i>client_secret</i>.<br />
<br />
Since OAuth <i>as popularly implemented</i> transfers users between multiple websites using a web browser, and that OAuth needs one website to send the other a <i>client_id</i> and <i>client_secret</i> (or equivalent), such as between AFCP and EVS, these are actually available to the user of the web browser if they monitor the browser's HTTP log. This is even possible in various custom browsers built into applications where a simple right click allows access to an inspector of some sort with network logging capabilities.<br />
<br />
In our case of AFCP utilizing EVS, this flaw would allow employees which have some access to AFCP to potentially gain more access than they should, and perhaps apply permissions they should not have access to. In a different example, if Facebook made use of an OAuth endpoint via a web browser for GMail where both the <i>client_id</i> and <i>client_secret</i> were transferred through the web browser, this would allow every user of Facebook to impersonate Facebook itself in this regard.<br />
<br />
This issue is present any time an OAuth endpoint expects to be sent the <i>client_secret</i> in plain text via a user’s web browser, or the API consumer is mislead into thinking doing so is a requirement and embeds a secret where they should not. A good indication that this vulnerability may be present is an endpoint where both the parameters for <i>client_secret</i> (or equivalent) and <i>redirect_uri</i> are expected (or even optionally allowed). The <i>redirect_uri</i> parameter is designed specifically for browser use to indicate to EVS where to send the user's browser after the login actions have been performed on the EVS side. As such, it means that if <i>redirect_uri</i> is in use for transferring on an endpoint, the flow is expected to be performed via the user’s web browser. Neither of the major <a href="https://tools.ietf.org/html/rfc5849">OAuth</a> <a href="https://tools.ietf.org/html/rfc6749">documents</a> specify or call for the use of such a case where both the parameters for <i>client_secret</i> and <i>redirect_uri</i> are expected to be used like this.<br />
<br />
A quick search online of such potentially offending OAuth-based APIs unfortunately shows many hits. Although Google offers many ways to use OAuth, <a href="https://developers.google.com/identity/protocols/OAuth2InstalledApp#handlingtheresponse">they have a flow which advertises the two being used together</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-U_YwQ6Pe5iM/VwG19Knj8wI/AAAAAAAAAQo/lI1EdgjxYrMuPjkacVx0gHeYGk9G0nUuw/s1600/google.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="287" src="https://1.bp.blogspot.com/-U_YwQ6Pe5iM/VwG19Knj8wI/AAAAAAAAAQo/lI1EdgjxYrMuPjkacVx0gHeYGk9G0nUuw/s320/google.png" width="320" /></a></div>
<br />
<a href="https://developers.podio.com/authentication/server_side">Citrix makes this mistake</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-QBTZHXSVhTU/VwG3w_L8K8I/AAAAAAAAAQ4/8ueRr0dyHM4MoYDZBhE-SSF_UDFctZ-0A/s1600/citrix1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="112" src="https://3.bp.blogspot.com/-QBTZHXSVhTU/VwG3w_L8K8I/AAAAAAAAAQ4/8ueRr0dyHM4MoYDZBhE-SSF_UDFctZ-0A/s320/citrix1.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-Mj7IOep1hQ0/VwG3fI1-ToI/AAAAAAAAAQ0/yUHMGKnQIHY4npV_9fMEGn6qaInpr9CjA/s1600/citrix1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
Along with <a href="https://developer.ciscospark.com/authentication.html">Cisco</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-HZ3pBF2b_U4/VwG6fwEhakI/AAAAAAAAARk/KAcd5lN9EPkO66h0UP7mvPptHhMgAoH7g/s1600/cisco.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://3.bp.blogspot.com/-HZ3pBF2b_U4/VwG6fwEhakI/AAAAAAAAARk/KAcd5lN9EPkO66h0UP7mvPptHhMgAoH7g/s320/cisco.png" width="320" /></a></div>
<br />
<br />
So does <a href="https://developer.github.com/v3/oauth/">Github</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-XDmbbq80vBY/VwG4I-FG-mI/AAAAAAAAARA/r_uIxIi67kIUTuUtTn5QD4l1H24JQuoUQ/s1600/github.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="261" src="https://2.bp.blogspot.com/-XDmbbq80vBY/VwG4I-FG-mI/AAAAAAAAARA/r_uIxIi67kIUTuUtTn5QD4l1H24JQuoUQ/s320/github.png" width="320" /></a></div>
<br />
And <a href="https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_understanding_web_server_oauth_flow.htm">Salesforce</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-R-P9VETXnEo/VwG4gVHyXCI/AAAAAAAAARI/YNJr6SYhdYM6VWvspMmAfQ1wUeGDRKSlg/s1600/salesforce.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="130" src="https://4.bp.blogspot.com/-R-P9VETXnEo/VwG4gVHyXCI/AAAAAAAAARI/YNJr6SYhdYM6VWvspMmAfQ1wUeGDRKSlg/s320/salesforce.png" width="320" /></a></div>
<br />
As well as <a href="https://buffer.com/developers/api/oauth">Buffer</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-x4AKHa8bvs0/VwG5B2LgDyI/AAAAAAAAARQ/YO3J9Ju88t4d3J-51hG1ydWdjHvRpv2hw/s1600/buffer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="261" src="https://3.bp.blogspot.com/-x4AKHa8bvs0/VwG5B2LgDyI/AAAAAAAAARQ/YO3J9Ju88t4d3J-51hG1ydWdjHvRpv2hw/s320/buffer.png" width="320" /></a></div>
<br />
<br />
And <a href="https://support.zendesk.com/hc/en-us/articles/203663836-Using-OAuth-authentication-with-your-application">Zendesk</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-bK2ur5AfUAI/VwG5d6woj3I/AAAAAAAAARU/Pi3dbl9PFN4roHq85zdx4twZnJJEjO3VQ/s1600/zendesk.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="314" src="https://2.bp.blogspot.com/-bK2ur5AfUAI/VwG5d6woj3I/AAAAAAAAARU/Pi3dbl9PFN4roHq85zdx4twZnJJEjO3VQ/s320/zendesk.png" width="320" /></a></div>
<br />
<br />
And the popular <a href="https://disqus.com/api/docs/auth/">Disqus</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-b-GD3JhY2zE/VwG55ELOAhI/AAAAAAAAARc/kQdImgBYA4Il4G1_vj5PP8ajYwExOp3Yg/s1600/disqus.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="164" src="https://1.bp.blogspot.com/-b-GD3JhY2zE/VwG55ELOAhI/AAAAAAAAARc/kQdImgBYA4Il4G1_vj5PP8ajYwExOp3Yg/s320/disqus.png" width="320" /></a></div>
<br />
I got this list after searching with Google for only two minutes. I'm sure readers can find many more with a little more effort. Note, the above list is not an indication that any of these services are directly vulnerable or that it's too easy to misuse them. Due to various factors, where for example, Zendesk specifically says their <i>redirect_uri</i> parameter is not actually used for redirection <i>in this particular case</i>, and that they even show that the endpoint should be called from an application using <a href="https://en.wikipedia.org/wiki/CURL">curl</a> - not a full fledged web browser, developers will hopefully not be misled into trying to do something dangerous with it. However, inexperienced developers in their applications may try to load one of these endpoints with a custom web browser. Furthermore, this combination simply being common in the wild is lowering developer's defenses against a potentially nasty misuse, where even the more experienced developers of OAuth-based services are blindly applying in similar situations, especially where the <i>client_secret</i> is named something else and the idea of <u>keeping a secret</u> is lost on them.<br />
<br />
A good indication that a service is broken in this regard is when several popular OAuth libraries fail to work with this service. Such services will generally offer their own "<a href="https://en.wikipedia.org/wiki/Software_development_kit">SDK</a>" which will work with them, and point third party developers to the official SDK if they complain their favorite library is unable to make use of their frankenstein-OAuth. These kinds of <i>customizations</i> often goes unnoticed, as the majority of developers prefer to make use of an advertised SDK when provided, and forgo rolling their own combination of software to utilize a service.<br />
<br />
This class of attack is labeled in the aforementioned security document as <a href="https://tools.ietf.org/html/rfc6819#section-4.1.1">4.1.1. Threat: Obtaining Client Secrets</a>. However, it doesn't even mention the specific attack case where a server is requiring use of a web browser for passing both the <i>client_id</i> and <i>client_secret</i> (or something of similar use). The authors of the document probably didn't expect anyone to design a service which could be this stupid nor developers using such APIs to make use of them with a custom web browser or SDK. These developers are mixing and matching separate components from different parts of the OAuth specifications in unspecified ways and expecting their platforms to remain secure without considering what new issues may be introduced by this patchwork approach. This is unfortunately the manner in which most of the OAuth players today function, and the already rampant problem only perpetuates itself as more and more providers jump on the bandwagon and copy the approaches they see <i>or <b>think</b> they see</i> being used by others.<br />
<br />
Odds are you'll be able to find many systems making use of the above services which are exploitable due to this problem. It's especially common in desktop applications, where a secret can be directly pulled out of a compiled application binary, even if the service being used isn't requiring anything insecure. It is important to note that Google offers many ways to use OAuth, and only one of them has an endpoint which receives both <i>client_secret</i> and <i>redirect_uri</i>. In the case of Google at least, they aren't recommending this endpoint for web browser based applications despite the presence of <i>redirect_uri</i>, but I'm sure that doesn't stop anybody from using it with a custom web browser or copying this flow into their service for an endpoint intended for regular browser use. In addition, Google appears to be the exception which proves the rule, and there's still the stupidity of all the other OAuth-based services out there which do not allow for secure OAuth-flows in such cases, as they require the <i>client_secret</i> (or equivalent) to always be passed, even when the flow is via a web browser. Worse, many of these services can <b>only</b> be used via a user’s web browser, a point which will be further elaborated upon below.<br />
<br />
The aforementioned security document mentions a couple of malicious things one can do once they steal an application's credentials (<i>client_id</i> and <i>client_secret</i>). I'll cover some issues below that can be coupled with this attack to allow one to do some nasty things which have not, to my knowledge, been previously covered. I'm sure my creative readers will also find additional ways to exploit stealing what is supposed to be kept <i>secret</i>.<br />
<br />
<h4>
</h4>
<h4>
Insecure Tokens</h4>
Token based authentication is a new concept to many developers. Therefore, it is also commonly misunderstood. Many developers designing something like EVS think if they simply follow some design guidelines (such as OAuth) on how APIs should work, or <a href="http://insanecoding.blogspot.com/2014/05/copying-code-copying-implementation.html">copy what other platforms are doing</a>, then their platform is inherently secure. However, in order for something to be secure, it requires that every single component be secure, that the combination of the components be secure, and that the overall framework be secure. Remember, security is only as strong as its weakest link, and it is not enough to rely solely upon adhering to some overall framework and assume that any and all use of it is therefore secure. The OAuth-based framework in and of itself provides very little in the way of ensuring the security of the underlying components (if it's not outright counter-secure for certain components).<br />
<br />
In order for a token-based system to have any semblance of security, the tokens generated must use a <a href="https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">cryptographically secure pseudo-random number generator</a>, <a href="http://insanecoding.blogspot.com/2014/05/dealing-with-randomness.html">a topic which is not so well understood</a>. Even worse, APIs for good CSPRNGs in popular scripting languages is severely lacking, yet such scripting languages are often the foundation of what is being used to design many popular modern services.<br />
<br />
If the tokens generated are predictable, that means that an attacker can impersonate users and perform malicious activities with their account simply by guessing at what a token might be. I saw one OAuth-based service from a major fortune 500 company which just uses some constantly incrementing ID (perhaps a database field?) for its tokens. I found another where I noticed the tokens generated all seemed to be the output of some monotonic-function. With further research, I found it was an extremely simple algorithm based on the current real world time. On these systems, I could log in as myself, see what the current token IDs are, predict what the next series are going to be, and then use that as part of a token exchange or other operation on behalf of the next random user. Combined with other techniques, even more targeted attacks can be accomplished.<br />
<br />
This class of attack is labeled in the aforementioned security document as <a href="https://tools.ietf.org/html/rfc6819#section-4.5.3">4.5.3. Threat: Obtaining Refresh Token by Online Guessing</a> and <a href="https://tools.ietf.org/html/rfc6819#section-4.6.3">4.6.3. Threat: Guessing Access Tokens</a>. While this issue is remediable, the amount of services currently making this mistake, and the ease with which this mistake is made, does not bode well for proving the security of a given OAuth-based service from an external review.<br />
<br />
Certain attacks against randomness, which is a whole other topic, can be used to utterly destroy an OAuth-based server if a security-hardened CSPRNG is not in use. While such issues are extremely problematic with other systems too, the issues are much more pronounced with popular OAuth implementations whose entire method of functioning is based around the concept of handing out random numbers. These tokens are generated server-side by EVS, and when used constantly as OAuth <i>as popularly implemented</i> does, drains the reliability of the server and increases predictability of all tokens involved. If you don't have a security-hardened CSPRNG for your environment which can protect against modern attacks, you're probably better off with another protocol which is more forgiving in this regard.<br />
<br />
It should however be noted that some OAuth-based implementations structure things in such a way to move randomness requirements to the client-side. While in many ways this is just moving a problem elsewhere, it does reduce the attacks surface from the service-side of things. This at least allows for more educated consumers of OAuth-based services to use a service they can trust in a secure manner, even if the less educated consumers may still be vulnerable. Applying this to our example, this setup would mean that the developers of AFCP can ensure its use of EVS is secure, and can't be exposed to such threats from EVS itself, even if ABC or XYZ do not use EVS securely.<br />
<br />
<h4>
Cross-site request forgeries</h4>
Before I get more into this one, let me just point out that despite the name, <a href="https://en.wikipedia.org/wiki/Cross-site_request_forgery">CSRF</a> attacks are not necessarily originated from a third party site. CSRF attacks can also be initiated from within sites themselves that allow users to post their own links, such as various online discussion and messaging software.<br />
<br />
There are many different techniques and frameworks designed to combat CSRF in various ways. OAuth-based integration can make many of these systems unusable or prompt various unsafe practices which can open sites up to attack.<br />
<br />
One defense mechanism against CSRF is to ensure that the <a href="https://en.wikipedia.org/wiki/HTTP_referer">referer</a> (sic) sent by the browser does not point to an external site. Since many OAuth implementations require that users are directed to certain pages from an external site, this defense cannot be enforced. Since the full scope of what pages and multitude of third party domains an OAuth server will perform redirection through, and since these URLs and domains involved are undocumented and may see periodic changes, the full scope of EVS domains and pages cannot be whitelisted.<br />
<br />
Also one needs to consider whether it’s possible for those offering EVS to turn around and try to attack AFCP. One of the principles behind OAuth is that OAuth-based services are not expected to trust their consumers, yet at the same time, require that the consumers fully trust them by allowing them blanket CSRF avoidance. An ideal authentication system would ensure a mutual level of distrust, not a one-way street.<br />
<br />
Partial whitelisting for either sources or destinations may also prove problematic. Depending on the anti-CSRF framework in use, tools for disabling certain features may be all-or-nothing, and it may not be possible to disable features on specific pages or for specific sources, forcing those who make use of EVS to stop using their anti-CSRF framework altogether.<br />
<br />
OAuth specifically defines the optional <i>state</i> parameter to be used to specify CSRF tokens to prevent CSRF attacks. However, I found that OAuth-based services commonly have limitations on length or allowed characters in <i>state</i>, and may not return it verbatim as required. Therefore, due to weird compatibility issues, many consumers of OAuth-based services find themselves forced to turn off CSRF protection altogether on redirection endpoints. This is labeled under <a href="https://tools.ietf.org/html/rfc6749#section-10.14">10.14. Code Injection and Input Validation</a>. Another consideration with the <i>state</i> parameter is that anyone with access to it on the EVS side can alter the request before sending the browser back to AFCP with an otherwise <i>valid</i> state parameter.<br />
<br />
OAuth-based API consumers are further limited by being required to define a URI or series of URIs up-front upon registration of their application or service, which is a white-list of URIs that can be used for <i>redirect_uri</i>. Putting aside for the moment major usability issues with such a system which will be discussed below, this limitation forces developers to start getting creative with the <i>state</i> parameter or other potentially dangerous ideas which can lead to a whole slew of issues. Many OAuth-based servers allow only a single white-listed URI, or require an exact match with <i>redirect_uri</i> and disallow additional parameters appended to it. This causes developers to stop using their anti-CSRF framework and start trying to shove all sorts of dangerous stuff into the <i>state</i> parameter or create other poorly thought out systems. The end result is some combination of <i>redirect_uri</i> and <i>state</i> that can be used to push users to some page they should not be pushed to. This is labeled under <a href="https://tools.ietf.org/html/rfc6749#section-10.15">10.15. Open Redirectors</a>.<br />
<br />
The problem with such redirection can potentially be exploited by itself, due to the combination of parameters not being fully authenticated, or this exploit can be combined with the issue - <i>Masquerading as an OAuth-using service</i> documented above which can be used to wreak havoc upon users. By making use of a stolen <i>client_id</i> and <i>client_secret</i>, a malicious party can create a redirection flow which appears genuine even to AFCP, by the fact that the authentication was performed with AFCP’s own credentials. A malicious user of AFCP may also be able to find a way to use or modify the <i>state</i> parameter, perhaps with stolen client credentials, in order to gain permissions they should not have within AFCP. All in all, due to poor design by OAuth <i>as popularly implemented</i>, and the difficulties external developers face with poor education on certain topics, attacking OAuth-based consumers is often much easier than it should be.<br />
<br />
Important reading here is also <a href="https://tools.ietf.org/html/rfc6819#section-3.5">3.5. Redirect URI</a>, <a href="https://tools.ietf.org/html/rfc6819#section-3.6">3.6. "state" Parameter</a>, and <a href="https://tools.ietf.org/html/rfc6819#section-4.4.1.8">4.4.1.8. Threat: CSRF Attack against redirect-uri</a>.<br />
<br />
<h4>
Section conclusion</h4>
In terms of security, OAuth <i>as popularly implemented</i> does quite poorly. OAuth fails to achieve many of the security objectives it supposedly sets out to achieve. Further, some OAuth-based services outright require that their consumers open themselves up to various attacks in order to use them. Even in cases where OAuth-based services can be used securely, which isn't always known (due to important service-side security details such as token generation methods being undocumented and closed source), OAuth still leads to many poor programming practices. OAuth does little to protect external developers, and various frameworks that such developers use in turn don't provide real security or can't be used securely without strict discipline and caution.<br />
<br />
This article in turn only covers some issues that I found rampant. Some of these issues are also the result of developers copying extremely poor practices they see others using alongside OAuth, practices which aren't even mandated by any OAuth specification.<br />
<br />
Developers for both OAuth-based services and the consumers thereof need to read and understand all of the linked documentation for implementing and using OAuth-based platforms. One also needs to fully comprehend the 50 classes of attacks listed, the multitude of attacks in each class, and note that <u>the implementation material and security guidelines are not exhaustive</u>. It should also be noted that this article only touches the surface of OAuth security problems, even if it does cover some issues not documented in the official material. Compounded with this, any changes made to the official OAuth proposals introduce a whole new set of security challenges, and these kinds of changes are unfortunately common. One in turn also needs to understand other security-related fields such as random number generation and security verification techniques to achieve any reasonable level of security with OAuth.<br />
<br />
If you're looking for real security, I recommend you look elsewhere. I'll cover some alternatives to OAuth in the final section.<br />
<br />
<h3>
</h3>
<h3 style="text-align: center;">
Usability Related</h3>
<h4>
Pull the plug architecture</h4>
OAuth <i>as popularly implemented</i>, requires that every integration done with an OAuth-based service require a set of application specific credentials to simply exist. These credentials aren't actually used to manage any particular set of user or organization accounts.<br />
<br />
This design allows EVS at any time to pull the plug (revoke the application credentials) on any application they no longer wish to service. This is often seen as an advantage to OAuth. However, this exact design is also telling of the intentions behind EVS's service, namely that they want full control over all use of EVS, and have the right to refuse service via APIs on a whim.<br />
<br />
If the company behind EVS decides to go into their own camcorder business, and wants to reduce competition, they can pull the plug on competing products that integrate with EVS. Imagine if desktop applications from their inception required approval from the Operating System vendor in order to function. When Microsoft decided to create Excel, they could have pulled the plug on Lotus 1-2-3. Nowadays they would be able to just prevent anyone from using Open Office or Libre Office, and force everyone who wants some desktop application office suite to purchase Microsoft Office.<br />
<br />
Besides being an evil way to practice business, this also hurts adoption of the service itself. Why would camcorder vendors want to waste money developing for EVS if any application they design can be shut down on a whim remotely? Even if they did offer some application with EVS compatibility, they certainly wouldn't want to advertise it for fear of being sued for false advertising and premature termination of service if EVS ever pulls the plug on them. Would you be comfortable providing a service subscription to someone, knowing that some third-party could come and completely disable the service at any time? Would you really expect that a client (especially a paying client) would accept an apology that says “sorry, EVS shut our service down, there’s nothing we can do”? Do you really think that such clients would let it go without seeking refunds and damages from you?<br />
<br />
As a manager making buying decisions and choosing a video hosting solution to purchase and integrate with AFCP, why would they choose EVS over other vendors, when EVS can quit working for their use-case at any time? When a vendor offers some service which does not use OAuth, they typically include some sort of API guarantee. Meaning their contract or terms of service include API usage specifically for the service package being purchased, and the package itself guarantees access to APIs. OAuth-based services, on the other hand, only guarantee access to the service directly, and tell prospective clients to review their OAuth APIs and apply for a <u>completely separate</u> set of developer access accounts if they wish to have API access. Due to this, the intelligent and discerning buyer will choose the non-OAuth-based service over an OAuth-based one every single time.<br />
<br />
A malicious user can also make use of the security issue above - <i>Masquerading as an OAuth-using service</i>, to acquire application specific credentials to impersonate it and do something which violates the terms of service for the APIs in question. This way attackers can ensure your application gets its plug pulled. In the case of our example, a competing camcorder vendor need only to obtain one copy of the camcorder to EVS application, extract the <i>client_id</i> and <i>client_secret</i> from the software, and then they can ensure that their competition fails.<br />
<br />
<br />
<h4>
Incorrect accounting</h4>
On top of a pull the plug architecture, OAuth-based services <i>as popularly implemented</i> will apply limits to how and when a particular application can make use of the service in question. For example, EVS might enforce that any specific application which uses EVS can only upload 100 videos in a 24-hour period. They may even charge the application vendor automatically for usage above that amount. As a result, if a camcorder bundles software to upload videos to EVS, their entire client-base can only upload 100 videos in a 24-hour period, even though each client has their own account with EVS. If one camcorder owner is particularly active one day and uploads a lot of videos to EVS, all other camcorder owners are now either locked out till the next day or will cause the camcorder vendor to be subject to additional fees. Worse, in the latter case, since the application performing the uploads is stand-alone without any intrinsic communication with any of the other upload applications, there is no way for the camcorder vendor to ensure that their limit is adhered to without significant overhead and a centralized tracking framework. Finally, many OAuth-based systems which do not change for overuse will simply pull the plug on third-party services which they see repeatedly going over their limits, holding the third-party service responsible for their users’ actions instead of terminating or charging the overactive users directly. This entire architecture is a scalability nightmare for third-party service providers, as every camcorder the vendor sells now brings them closer to the point where EVS might shut them down and destroy their business.<br />
<br />
Most non-OAuth-based services usually have normal accounting in place where limits of the service are applied to each individual user or organization of users (depending on whether the service is for personal-use or enterprise-level, respectively). OAuth <i>as popularly implemented</i>, however, groups together completely unrelated users and organizations, and makes all of them subject to the whims of one.<br />
<br />
A “simple” workaround for incorrect accounting is to require each user or organization who wants to use some application with EVS to go apply for their own developer account. This, though, has the unfortunate consequence of forcing every camcorder buyer to figure out for themselves how to obtain a developer account, a process which is often far from simple. Then the user will have to figure out how to set up the camcorder software to use that account, increasing the possibility of human error.<br />
<br />
I took the above described approach with an application I wrote about a year ago which integrates with a Google service, and every first-time user of it has to log into the Google Developer Console and perform 15-20 steps before they can use the application. Besides being annoying, I found I couldn't even give clear guidelines on how to perform these steps, as Google has completely restructured their developer console and renamed things half a dozen times in the past year. Even the first time I wrote a manual for navigating the console and selecting the right options, the manual became obsolete the very day after I published it. I can't point to any documentation on Google's side either, because they have no A to Z documentation specifically for obtaining the specific kind of OAuth credentials necessary for the application to work with the service in question.<br />
<br />
The sane approach as used by non-OAuth-based services would be to allow every user or organization account manager (again, depending on whether the service is enterprise-level or not) to have a quick and easy to use control panel which can generate API access tokens for any service which is part of the account, with whatever permissions are needed. However OAuth <i>as popularly implemented</i> never seems to do this. Some OAuth-based services I've seen even require manual approval every time a user applies for a developer account. All this means using OAuth <i>as popularly implemented</i> creates considerable extra burden on the support staff for EVS, the camcorder vendor, and all of their users.<br />
<br />
<h4>
</h4>
<h4>
URI lock-in / Application incompatibility</h4>
OAuth <i>as popularly implemented</i> requires that a URI be provided up front to EVS when registering for a developer account or application credentials to be used as part of the authentication process in the new application. Remember, this URI will be used <b>every time</b> a user in the third-party service authenticates with the OAuth-based service. After the user logs in, the EVS special login pages will direct the user's browser to the specified URI with the tokens required to access the APIs. Some OAuth-based services will only allow for a single URI up front, while some other will allow several URIs to be entered. In the latter case, when the third-party service initially directs the user's browser to EVS's special OAuth login page, it can also specify which of previously enumerated return URIs to use.<br />
<br />
If only one URI is allowed, or the amount of allowed URIs is not enough, often developers will find themselves being forced into opening themselves up to attacks discussed above under <i>Cross-site request forgeries</i>. But this is only the beginning of the issues this requirement causes.<br />
<br />
OAuth <i>as popularly implemented</i> also tends to think that every use of it is by a web application built using the <a href="https://en.wikipedia.org/wiki/Software_as_a_service">software as a service</a> (SaaS) model. If some product, say AFCP, is installed <a href="https://en.wikipedia.org/wiki/On-premises_software">on-premise</a> and deployed on each individual client's personal domain, what URIs do the vendors of AFCP provide when registering AFCP for EVS? This is even more ridiculous when the application using EVS is a desktop application like the camcorder software. Such software has no associated website at all! How are these applications supposed to work with these OAuth-based APIs?<br />
<br />
It should be noted that getting a plain HTTP client, such as <a href="https://en.wikipedia.org/wiki/CURL">cURL</a>, is readily
available in every major programming language, and can be used in any
kind of application. Web browsers with modern features cannot be used in
non-graphical clients, such as command-line/terminal applications, and
are only available in some programming languages. This means that OAuth-based APIs are severely limited in what kinds of applications they can be used with, making many desired use-cases impossible. It should be noted that many of the popular OAuth based services also require JavaScript on either their special OAuth login pages, or as part of the subsequent redirect process, further complicating simple usage in an application.<br />
<br />
Even for SaaS products, if it's deployed on regionally diverse domains (mysaas.us, mysaas.co.uk, etc…) and EVS does not allow for entering enough URIs, then what? What about situations where every client gets their own sub-directory or sub-domain when using the service, how does one pre-register these or deal with the case when the amount of URIs required exceeds the amount EVS allows?<br />
<br />
Even if EVS is a service which allows post-editing of the URI white-list, and allows for a nearly unlimited amount of URIs to be entered, how does the list get updated exactly? I have not seen a single OAuth-based service which provides any meta-APIs for modifying the URI white-list. In today's day and age, most SaaS deployments are or should be automated. When a client registers for a service, credit cards can be automatically processed, servers automatically deployed, sub-domains automatically added to <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a>, initial credentials automatically e-mailed to the administrator, and so on. But how do the new URIs get added for any OAuth-based services used by the just deployed SaaS instance? Manually entry is utterly unacceptable and completely unscalable.<br />
<br />
The workaround most suggest is to have some website set up which handles redirection for all clients and users. However, besides the aforementioned security issues when not done carefully, this can get expensive fast. In order to have all requests processed by a fixed location, enough load balanced infrastructure needs to be set up to handle the load, even though it could otherwise have been handled by the same server or servers which handle the rest of the service for the given client. This also binds on-premise and desktop applications to this extra website, and will be unavailable if the extra website goes down for some reason. This has now become an annoying and costly dependency solely to manage a misfeature.<br />
<br />
Another workaround suggested for desktop applications that don't want to be bound to a website is to build in an HTTP server into the application, and set the return URI to <i>http://localhost/</i>. It's bad enough that a web browser - often a full featured one with JavaScript support - needs to be built into the desktop application just to handle the login process, now they recommend a web server should be built-in too! I should point out, though, that whitelisting <i>http://localhost/</i> is not enough, as that URL implies the server is running on port 80. Not every application can grab that port, nor is the port always available, as something else may be using it. To make a robust application that can handle every eventuality as best as possible, one also needs to whitelist <i>http://localhost:1/</i>, <i>http://localhost:2/</i>, ..., <i>http://localhost:65534/</i>, <i>http://localhost:65535/</i>. This probably sounds ridiculous to you, and that's because it is. This also doesn't even account for the fact that some security software may get cranky or scare users if it finds an application is launching a server.<br />
<br />
Being faced with this and some of the previous issues and finding only poor solutions offered, I came up with a completely different solution which I use with some OAuth-based services. I created a small web application that I can throw up on any server which asks in a web-based form to be populated with a <i>client_id</i>, <i>client_secret</i>, and anything else the OAuth-based service requires from what it thinks is an application. After submitting the details, the application redirects the user to the special OAuth login page, and after redirected back, the application displays whatever tokens it just got. The user can then manually paste these tokens into the real application they want to use which does not need to be tied down to any site nor requires browsers, servers, and graphical interfaces be used by the real application.<br />
<br />
My above solution first requires users register for their own application or developer account, whitelist whatever the URI is to the small web application, paste their data into the small web application, log in, get the tokens, and paste those into the real application. It works well, circumvents any needs for a web browser and <i>redirect_uri</i>, and it's only a mere 20-25 steps for every user to perform. This would be much simpler for the users if the OAuth-based service itself showed them the courtesy of just supplying a page in their account where they can generate tokens with whatever permissions they wanted. Instead, these services would rather force third party developers to provide such finery, and make their users lose their minds.<br />
<br />
<h4>
</h4>
<h4>
Open-source unfriendly</h4>
A minor point to the above, OAuth is also unfriendly to open source applications, because every user making use of it is required to register their own developer account in addition to whatever user account they already have with a service. It would be unsafe to include credentials in the source or within binary packages which can be stolen, see <i>Masquerading as an OAuth-using service</i> above. This is on top of all the previous remarks about not knowing what URI to use, incorrect accounting, and all the other limitations.<br />
<br />
<h4>
Low availability</h4>
Another major flaw with OAuth <i>as popularly implemented</i> is its low availability (which is the opposite of <a href="https://en.wikipedia.org/wiki/High_availability">high availability</a>). OAuth-based systems differ as to the exact details, but they usually expire the tokens they provide that are required to be used with the actual API calls. Some of them allow a way to refresh them, but usually only for a limited amount of time. These systems require that the user of an account be present to re-authenticate with the service like clockwork. Besides being annoying, requiring re-authentication in middle of an operation and being able to continue the process after doing so may cause the design of the platform to fail in ways described above under the issue <i>Cross-site request forgeries</i>.<br />
<br />
<br />
Now in the case of the camcorder software, where the user was using the provided software to upload to EVS for a while, what happens when the tokens expire? Say the user queued up a bunch of videos and then went camping for the weekend. They come home to find the first few videos uploaded, and then the rest failed because they were not around to re-authenticate. This user will be extremely annoyed, especially if they promised someone the videos would be up by a certain time before they left.<br />
<br />
When we consider our usage scenario with AFCP, this requirement to re-authenticate gets even worse. Normally when an administrator assigns someone new to a department, AFCP runs through the list of all integrated users in that department, and adds permission for this new user to access the EVS videos of the other users in the same department. However, if one of those previously existing users needs to re-authenticate, AFCP cannot automatically assign permissions for accessing videos to the newly added user. If the videos of the particular user who needs to re-authenticate is crucial for training a new employee, and the user is on some extended leave (vacation, maternity, etc...), the new employee cannot even be trained.<br />
<br />
This expiratory handicap can make sense in various usage scenarios, but completely destroys what EVS integration with AFCP is trying to accomplish. If your usage scenario is AFCP, you really need to look for a service to use which does not use expiring tokens, which OAuth <i>as popularly implemented</i> unfortunately does. However, I did come up with a workaround which I use with some of my applications.<br />
<br />
If a platform needs to re-authenticate on behalf of a user when they're not around, one would need their credentials to do so. I've exploited the flaw described above - <i>Stealing credentials / gaining elevated Access</i>, in which I prompt users for their credentials when they log in, store them (encrypted of course), and reuse their credentials automatically when tokens expire. This technique works as long as the user does not change their password and additional authentication factors are not in play.<br />
<br />
I created a SaaS application which exploits this flaw over two years ago. Several thousands of users have since willingly entered their credentials into the application, and not a single one of them has ever complained or noticed anything suspicious. Better yet, the vendor behind the OAuth-based service I built upon heard about my application from some of their users, and was pleased as to how they gained access to a larger user base. They contacted my employer and asked if we could provide them with some accounts on our platform for their sales department to show off to prospective clients, as well as a few other key personnel. We did so, and have since captured the credentials of several of their high ranking employees, including an executive responsible for managing their OAuth-based service. Not one of them noticed anything wrong with what we were doing, which goes to show that this is a viable workaround - however unorthodox.<br />
<br />
While the above technique can turn a useless OAuth service into something profitable, it'd be better if it was achievable without needing to undermine security, defeat the few benefits of OAuth, and rely on exploitable flaws. I'll discuss below what really should be done instead of trying to always limit usage to the point that a service becomes unusable.<br />
<br />
<h4>
Not enterprise-ready</h4>
All the above usability issue sections show that OAuth <i>as popularly implemented</i> is not ready for the enterprise market. Contracts with OAuth based service providers do not actually cover the services you want, accounting is incorrect for organizations making them subject to unrelated third parties, there's extra setup annoyance, it’s not scalable, and the services repeatedly fail to work without manual user intervention from specific users. All this on top of not necessarily being able to conform with an organization’s security requirements.<br />
<br />
However a real problem with OAuth-based services is that their APIs always seem to be geared towards only managing things for a single user as themselves. When someone personally gets an account for a service, they obviously do not want others to be able to take control of their account. But when an organization at the enterprise level purchases many accounts for their employees company-wide, they expect to have full control over all accounts, there is no room for individuals. In enterprise scenarios, no AFCP user should even be able to opt-out of the EVS usage, as the higher-ups want all resources within a department to be accessible to the entire department. Therefore, in order to be feasible for an enterprise, whoever or whatever is in charge of administrating all these accounts needs to be able to perform any action on behalf of any user. If this requirement is not met, the enterprise company will need to find a different service which does make this possible.<br />
<br />
I know of several OAuth-based services which in their front-ends for non-API usage allows for companies to set up authentication to their services using <a href="https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language">SAML</a>. With SAML, any login request signed by a particular <a href="https://en.wikipedia.org/wiki/Public-key_cryptography">private key</a> is able to gain entry to a platform as the user who is identified in the signed request. Whoever is in charge of managing the SAML <a href="https://en.wikipedia.org/wiki/Identity_provider">identity provider</a> and its private keys, usually the IT staff at the companies, can masquerade as any user they please and perform needed activities on their behalf.<br />
<br />
What companies really want when they purchase a package with a service is to use an automated system to perform needed activities as required on behalf of their personnel. The APIs of the OAuth-based services, however, don't provide a way for any user - even at the administrator level - to perform activities on behalf of other users, nor do they provide a way to acquire tokens representing those lower-level users to use with the APIs on their behalf. An administrator cannot assign permissions for any videos in EVS except what they themselves have uploaded. When I asked developers of these services why not, they responded that providing such a feature would weaken the security of their platform. Yet these services which support SAML <b>already</b> have their security “weakened” in this way, though I fail to see how allowing administrators to effectively administrate their users, a situation which is usually considered desirable, can be called a “weakness”.<br />
<br />
All in all, we see that the typical service with APIs built upon the foundation of OAuth <i>as popularly implemented</i> is designed and maintained with myopia regarding what enterprise customers actually want to achieve with a system. These APIs generally don't allow for anything to be done which couldn't already be done the same way with their existing user interface. However if you can't build anything that wasn't directly achievable with the provided interface, why bother offering APIs at all? Providing APIs is not so that the exact same UI with the same limited feature-set can be recreated over and over in other contexts, since today with <a href="https://en.wikipedia.org/wiki/Single_sign-on">SSO</a> technology to access a website (such as SAML), there's no real point in that. The point in providing APIs is so that something new can be created which can offer added value to a service. If it's not possible to do more with the APIs than what is already built-in, then the APIs are of little value and no self-respecting enterprise is going to take the product seriously.<br />
<br />
Don't get me wrong, with limited APIs in place, there are still those making third party applications. However these applications are generally just embedding a subset of a service’s existing features into a different context with no real added value. If a third-party application is actually providing something with added value with these crippled APIs, they end up also requiring users to perform dozens of steps and tons of copying and pasting to accomplish even the simplest of tasks. In such a situation, odds are a discerning buyer is not going to consider buying that product for all personnel, especially when there are alternatives available which aren't such a pain to use. A good rule of thumb, bad integrations from otherwise competent developers are the result of bad APIs. Therefore, if all of the so called “integrations” being created for a service either lack added value or are extremely clunky, it’s probably an indication that the APIs are lacking what enterprises need. If your product lacks what enterprises need, you can expect that most enterprises will not be purchasing your product.<br />
<br />
It should be noted that the only major services which have thriving OAuth-based ecosystems are platforms designed specifically for individual user use, and are not geared for enterprise use-cases. Facebook and Twitter (both of whom were in part responsible for the formation and adoption of OAuth), have a huge repertoire of apps making use of them. Google is another interesting example. Several applications, platforms, and services exist which make use of Google’s OAuth APIs, but all of these are noticeably geared towards individual Google accounts, despite the fact that Google has a large business offering as well. In contrast, Google Apps for Business services generally offer alternative systems and protocols which are more appropriate for the enterprise market than OAuth, as they recognize their OAuth-based APIs to be incapable of providing the needed functionality.<br />
<br />
<h4>
Section conclusion</h4>
While the security of OAuth <i>as popularly implemented</i> is not all that it's considered to be, the worst part about it is how crippling it is to actually work with. Many of the things OAuth sets out to accomplish are actually detrimental, especially in enterprise settings, and at best, you'll need to find ridiculous workarounds to even get any use at all out of a service whose APIs are built upon OAuth.<br />
<br />
The following is a mostly accurate depiction from one of my colleagues as to where the world with OAuth is heading:<br />
<blockquote class="tr_bq">
<br />
Allow me to present an alternative to OAuth which provides the same level of functionality and security but is far simpler to implement. We call it <i>NoAuth</i>. The protocol is as follows: No one may ever authenticate, period. Simple, effective, and we estimate that it will take developers a mere fraction of the implementation time of OAuth, cutting your costs significantly. <i>“NoAuth, because the future <u>deserves</u> absolute impenetrability.”</i></blockquote>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-8zjIUqeP4gs/VwZUY8XnO5I/AAAAAAAAASY/s-gEf_sEV2IBatKt7ZBSAqAaKD7kjuJvA/s1600/no-oauth-dilbert-like.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="132" src="https://1.bp.blogspot.com/-8zjIUqeP4gs/VwZUY8XnO5I/AAAAAAAAASY/s-gEf_sEV2IBatKt7ZBSAqAaKD7kjuJvA/s640/no-oauth-dilbert-like.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-2gXYWFg7ndE/VwWa2W4lQSI/AAAAAAAAASI/SinxsyeVluMJolTpjIbAkvfi0vKTR9jow/s1600/no-oauth-dilbert-like.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
A key misdesign of OAuth <i>as popularly implemented</i> is thinking that all external services should be mistrusted equally. Let administrators define what should and shouldn't be done with a service, and what capabilities should be made available with various API tokens. Don't make this decision for them and limit what can be done with a service in its entirety just because there can be some rogue applications out there.<br />
<br />
For applications I provide which make use of API services, I frequently get clients who ask me what exactly is my system doing that it needs certain permissions, and that's okay. Administrators are capable of deciding what they want or don't want to allow and what the ramifications may be.<br />
<br />
My employer is commonly approached by third parties asking us to integrate some service with another service, telling us how much revenue we'll pull from sales if we partner with them. Every time we find out one of these services is using OAuth we groan, because almost always, it means the project is infeasible. Yet the third party thinks there must be some magical way to do it because we've done some amazing things with other third parties which are not using OAuth. When we ask them to improve their APIs or drop this ridiculousness with OAuth, they claim they can't because <i>this is how it has to be done</i>.<br />
<br />
More often than not, we find whatever is desired really is not feasible, and the workarounds are either not possible or too ridiculous, time-consuming, and annoying to the end-user for the case at hand. These third parties beg us to then <i>just do something with it</i>, as if that were a sane business practice. We're not going to create some product for which there's no business case and doesn't look like it'll break even, let alone make a profit.<br />
<br />
I've seen a number of these projects come and go over the years. Usually the third party can't find anyone dumb enough to waste their time and effort on working with them, although on occasion they do. The outcome is always something which doesn't sell well, and the third party throws in the towel a couple of months later.<br />
<br />
Even large enterprise organizations are coming up with some new product every year that they then try to force on the market, but time after time, these never seem to go anywhere. Despite the fact that the product has no real use, and other businesses can find no way to make it useful, they blame everything but a crucial issue at the heart of the matter. Instead of fixing the actual flaws preventing integration, the lifeblood of enterprise adoption and retention, they think it must something else, discontinue the product, and try releasing a different but similarly crippled product the next year.<br />
<br />
<h2 style="text-align: center;">
Alternatives to OAuth <i>as popularly implemented</i></h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-LybF43duWaY/VwKA3VUW6tI/AAAAAAAAAR4/J4i6JewKgyo8H3CwzFuKWKDNzfXqmDWRA/s1600/no-oauth.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-LybF43duWaY/VwKA3VUW6tI/AAAAAAAAAR4/J4i6JewKgyo8H3CwzFuKWKDNzfXqmDWRA/s1600/no-oauth.png" /></a></div>
<h3 style="text-align: center;">
What do proper OAuth-based designs look like? </h3>
Now that we've seen that OAuth <i>as popularly implemented</i> is utterly broken, when does OAuth actually work?<br />
<br />
OAuth systems that adhere mostly to the earlier OAuth specifications and concepts are generally more secure and less broken than those based on later specifications. This is not to say that all OAuth 1.0 implementations are secure, but they're usually less problematic. These systems will normally follow one of two distinct approaches:<br />
<ul>
<li>By providing users the ability to directly generate access tokens in their account which they can dole out to software they wish to integrate, a lot of the silly requirements and insecure transfers and redirects are bypassed.</li>
<li>By using a system where the <i>client_secret</i> does not function as a piece of data which is passed verbatim like a common password, but is instead itself a <a href="https://en.wikipedia.org/wiki/Shared_secret">cryptographic shared secret</a> used to generate <a href="https://en.wikipedia.org/wiki/Message_authentication_code">authentication codes</a>, the remainder of the token system, silly requirements, and insecure transfers and redirects are bypassed. In these systems, users generally do not even have their own credentials and only use SSO mechanisms.</li>
</ul>
Precious few OAuth-based systems are designed like this though, and these systems generally look nothing like OAuth as used everywhere else. Since these systems stick closer to the OAuth 1.0 specifications, which is officially deprecated, systems using this approach eventually get <i>"updated"</i> to be restructured with all sorts of OAuth 2.0 concepts and additions, thereby ruining their security and usability. This is a reason I'm hesitant to condone anything OAuth based, because even if using an earlier, more operable style of OAuth, some manager is going to have the bright idea that the system will need <i>"improving"</i> and will break it. Better to use something else altogether than begging for problems.<br />
<br />
<h3 style="text-align: center;">
Other options</h3>
Looking for other options, people often want to know what other <i>frameworks</i> are out there. However, one does not need some framework to achieve a well understood and secure design. As is, every service has their own take on what OAuth looks like, so there really is no exact approach on how authorization works anyway. Hunting for a framework is often over-complexifying what can be done simply. The only really difficult component that requires a good specification is how to sign variables to prevent tampering with key parameters used, and most OAuth-based implementations do nothing of the sort anyway.<br />
<br />
The largest provider of web services on the market today is <a href="http://aws.amazon.com/">Amazon</a>. They are the premier provider for enterprises the world over, and utterly dwarf everyone else with a whopping greater than 30% combined market share. Amazon's approach is to provide all their accounts and account administrators with access to a control panel where they can generate application credentials. These credentials can be specified as to what Amazon services they can work with, what actions they can perform in those services, and what permissions they have to work on. These credentials can be revoked by the <u>account holder</u> at any time if necessary.<br />
<br />
Amazon's authentication and authorization techniques for their APIs do not require any redirection methods which are inherently limiting and potentially unsafe. <a href="http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">Amazon's protocol</a> never sends along any credentials directly, rather they're used for signing data, and can ensure parameters remain tamper-proof if there's any need to send them through a browser.<br />
<br />
Amazon's design has proper usage accounting, includes API usage as part of every account, and all API authentication and authorization is initiated from the Amazon side with the creation of application credentials from <b>their</b> control panel. These credentials are then used directly in the API process without any sort of additional token exchange system. This design achieves all the real security objectives OAuth <i>as popularly implemented</i> achieves, and it also avoids all the security and usability issues enumerated above.<br />
<br />
One downside I have to say about Amazon is that their permission system is somewhat confusing and not all that user friendly. This happens to be true though of most control panels, and in any case, is a matter of user interface design and is not a mark against the authorization process itself. Also, Amazon's control panel can be navigated fairly quickly and even be used with their APIs, unlike, say, Google's, which, as far as I know, has no meta-APIs, and requires at least a dozen steps to do anything with it.<br />
<br />
Amazon's authentication and authorization method is also copied by several other service providers on the market. Google themselves even allow for it in certain of their enterprise products. Google themselves also acknowledge that a pure OAuth design is poorly suited towards enterprise services, and for their enterprise services, they recommend the use of <a href="https://en.wikipedia.org/wiki/JSON_Web_Token">JSON Web Tokens</a>.<br />
<br />
JWT is a <a href="https://tools.ietf.org/html/rfc7519">specification</a> for allowing SSO or API usage between services. In many ways JWT is like SAML, although unlike SAML which is confusing, built upon XML Security (which is anything but secure), and not geared for API use, JWT achieves the primary SAML objectives in a simple and easy to use manner without all the headaches. If you have an <a href="https://en.wikipedia.org/wiki/Hash-based_message_authentication_code">HMAC</a> implementation and know how to structure and parse JSON, then you can use JWT. For those of you who want something off the shelf, there are <a href="https://jwt.io/">plenty of JWT libraries already available</a>.<br />
<br />
Google’s use of JWT is more advanced than typical though, and instead of HMAC, they require the use of <a href="https://en.wikipedia.org/wiki/RSA_%28cryptosystem%29">RSA</a> <a href="https://en.wikipedia.org/wiki/Digital_signature">digital signatures</a> which is a more advanced, and less popular concept than HMAC in this area. Google's control panel allows account administrators to generate a new key-pair for their enterprise services, and download the private key to use for signing API log-ins. While this is more secure than HMAC, Google really over-complexifies the whole process, not to mention that they completely redesign their control panel frequently, making it confusing to repeatedly use. I recommend looking instead at what others are doing with JWT if you need examples.<br />
<br />
Another technique being used is that of services allowing definition of what kind of permissions third parties need in some kind of XML or JSON file that they can post on a web site. Users can then visit a page in their account where they can paste the URL to this file (or paste the contents of it), and the service will display a list of what kind of permissions the external service or application wants to use and any descriptive information it may contain. If users want to approve this, they can generate credentials which they can then paste into the third party application or service they are using. Users can later revoke the credentials if they want to disable the third party product. This is also a very secure design which doesn't have any ridiculous burdens placed on developers, includes API services for all accounts, provides permissions, and initiates the flow with the service itself, not the third party.<br />
<br />
All you really need from the service-side in order to manage authorization is some panel which allows users with the appropriate role (administrator or account owner) to generate API usage credentials which each have permissions and a possible expiration (if desired) assigned to them. These credentials can then be used over any secure authentication system you choose, such as something simple like <a href="https://en.wikipedia.org/wiki/Basic_access_authentication">HTTP Basic Authentication</a> over <a href="https://en.wikipedia.org/wiki/HTTPS">HTTPS</a>, which is available with practically every HTTP library available, <a href="https://en.wikipedia.org/wiki/Digest_access_authentication">HTTP Digest Authentication</a>, which is more secure and supported by most high-grade libraries, or something else based on authentication codes utilizing a cryptographic technology which does not require any credentials ever being passed over the network, such as HMAC, RSA, or <a href="https://en.wikipedia.org/wiki/Elliptic_curve_cryptography">Elliptic Curves</a>. HMAC in particular is already in use by nearly everyone implementing some form of authorization or authentication (including Amazon and even some OAuth implementations).<br />
<br />
These various well established techniques decrease the burden of needing to study the effects of one framework’s interaction on others, such as those for anti-CSRF, in order to create a secure platform, and can generally be implemented in a modular manner which can simply be dropped into an existing architecture. They avoid the possibility for stealing the user's or the application's credentials. They also don't require any constant use of elaborate CSPRNGs. These kinds of systems existed long before OAuth and are popular today too. OAuth may have better security than some poorly designed systems which require users’ personal credentials and have other weaknesses, but OAuth is not a substitute for the real designs that already existed. The problems that OAuth claims to solve do not actually exist in the existing well-designed systems, and OAuth <i>as popularly implemented</i> actually introduces many of the problems it claims to solve, along with several others which never existed in the first place. Despite the hype, OAuth does not inherently provide amazing security, and due to the numerous drawbacks and implementation difficulties, the other well-thought-out options are vastly better.<br />
<br />
If you're going to be designing a service and provide API access, please, really think about what you're trying to achieve, and don't just copy what you see others do or buy into ridiculous hype. If you must copy someone, try to copy Amazon (the best), <a href="http://docs.rackspace.com/">Rackspace</a>, <a href="http://sldn.softlayer.com/article/Authenticating-SoftLayer-API">IBM SoftLayer</a>, <a href="https://www.linode.com/?r=9cdf9e637dcff3ba9792eef8044009e51da50d79">Linode</a>, <a href="https://www.vultr.com/api/">VULTR</a>, <a href="https://www.zoho.com/crm/help/api/using-authentication-token.html">Zoho</a>, <a href="https://support.zoom.us/hc/en-us">Zoom</a>, or others who seem to currently have some inkling on how to structure a straight-forward and sound authentication system for APIs.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com343tag:blogger.com,1999:blog-833174317742362874.post-81009066140024003882014-12-13T11:22:00.002-08:002014-12-13T11:57:36.139-08:00Effective Modern C++ Quick Review and SaleFor those of you familiar with Scott Meyers amazing <a href="http://www.amazon.com/gp/product/0321334876/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321334876&linkCode=as2&tag=insacodi-20&linkId=RJZO5T45DD7IO6OK">Effective C++</a> and <a href="http://www.amazon.com/gp/product/0201749629/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0201749629&linkCode=as2&tag=insacodi-20&linkId=EFC3JYVEX4G6RXT6">Effective STL</a> (both of which I highly recommend to C++ programmers), <a href="http://www.amazon.com/gp/product/1491903996/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1491903996&linkCode=as2&tag=insacodi-20&linkId=TISAZRSYJ5SDOIVH">Effective Modern C++</a> is now out which covers best practices for new features introduced in C++ 2011, as well as some upcoming features of C++ 2014.<br />
<br />
<b>For one more day only, you can also get $10 off any of the above links to Amazon if you use the code <a href="http://www.amazon.com/b?ie=UTF8&node=10510039011"><u>BOOKDEAL25</u></a> when you checkout</b> (if your order totals $40 or more).<br />
<br />
For those of you unfamiliar with the series, I'll provide you with a quick review.<br />
<br />
Effective C++ is an excellent book for C++ beginners and experts alike. It helps beginners understand what is and how to choose the best tool for the job when C++ offers multiple ways of doing things. It can also help even experts to better understand how the language works, and what is and isn't good practice. You'll learn how to correctly navigate this large language, and end up writing programs where memory management or other resource issues are a thing of the past. You'll learn how to design programming interfaces that look sane, and you won't come back to them later asking yourself what drugs you were on when you originally wrote the code.<br />
<br />
Effective STL dives more into the standard library of C++, helping the developer understand which container or algorithm is the best tool for the job, and how to use them most effectively. If you've ever wondered to yourself if you should use an std::vector or std::list, you need to read this book. If you've ever wondered what <i>functors</i> are or why bother with features which at first glance may seem like a roundabout way of doing things, you'll unlock the true potential of these features and more, and learn how what may seem weird to the untrained eye is actually powerful and produces better code.<br />
<br />
Effective Modern C++ introduces the reader to new features added to the language in recent years, and as the other books in the series, helps them to better understand them, get the best out of them, and use them correctly. You'll learn how to choose and make the best out of the new resource management introduced with std::shared_ptr, std::unique_ptr, and std::weak_ptr, avoid common pitfalls with the new threading library, new functional programming techniques exposed in modern C++, as well as how to achieve even more optimization with all the new features.<br />
<br />
The series also includes <a href="http://www.amazon.com/gp/product/020163371X/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=020163371X&linkCode=as2&tag=insacodi-20&linkId=DVAEYMWN5VLW5HYQ">More Effective C++</a>, however I do not recommend this book. Unlike the previously mentioned books, the latest edition of More Effective C++ came out before C++ was even standardized, and has been out of date for well over a decade. It does explain certain features of the language, most of which are still present today, but those features are best covered by an introduction to the language itself like <a href="http://www.amazon.com/gp/product/0321563840/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321563840&linkCode=as2&tag=insacodi-20&linkId=ZPEGNDHFCXYZHXLH">The C++ Programming Language</a> (<a href="http://www.amazon.com/gp/product/0321958322/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321958322&linkCode=as2&tag=insacodi-20&linkId=DAEFIGVAVDYRESBB">hardcover</a>), <a href="http://www.amazon.com/gp/product/0321992784/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321992784&linkCode=as2&tag=insacodi-20&linkId=DCBZRZ5WNWVS7QDD">Principals and Practice Using C++</a>, and <span style="font-family: inherit;"><span style="font-weight: normal;"><span style="font-size: small;"><span class="a-size-large" id="productTitle"><a href="http://www.amazon.com/gp/product/0321623215/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321623215&linkCode=as2&tag=insacodi-20&linkId=JDNEFMB3GE4T4POD">The C++ Standard Library</a>. One thing More Effective C++ covers that isn't in these great C++ learning and reference books is how to use C and C++ together effectively. Although in my opinion that's better covered in the more C focused book <a href="http://www.amazon.com/gp/product/0131774298/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131774298&linkCode=as2&tag=insacodi-20&linkId=2MYCG23WLFRNJRXK">Expert C Programming: Deep C Secrets</a>.</span></span></span></span><br />
<span style="font-family: inherit;"><span style="font-weight: normal;"><span style="font-size: small;"><span class="a-size-large" id="productTitle"><br /></span></span></span></span>
<span style="font-family: inherit;"><span style="font-weight: normal;"><span style="font-size: small;"><span class="a-size-large" id="productTitle">For those of you thinking to yourself that C++ is a horrible language, especially since it has so many books printed for it on how to correctly use it and get the best out of it, you might want to realize that most well established languages have such books. For example, <a href="http://www.amazon.com/gp/product/0321356683/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321356683&linkCode=as2&tag=insacodi-20&linkId=2TY2WTSDIAWS4SK5">Effective Java</a>, <a href="http://www.amazon.com/gp/product/0321812182/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0321812182&linkCode=as2&tag=insacodi-20&linkId=GLLP7NZHS4UBI6QF">Effective JavaScript</a>, and the groundbreaking <a href="http://www.amazon.com/gp/product/0596517742/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0596517742&linkCode=as2&tag=insacodi-20&linkId=6MHU3L7QA2C53ZNN">JavaScript: The Good Parts</a>. Proper programming design is an art, and one could even go as far to say that if a book of this nature does not exist for a language, it's because no one has yet figured out how to really use it effectively.</span></span></span></span>insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com57tag:blogger.com,1999:blog-833174317742362874.post-12526207540305553472014-10-02T11:37:00.001-07:002014-10-02T12:15:58.117-07:00What's really wrong with systemd?Unless you've been living under a rock or on a park bench during the past two years, you've probably heard of <a href="http://en.wikipedia.org/wiki/Systemd">systemd</a> and the <a href="http://ewontfix.com/14/">numerous</a> <a href="http://www.networkworld.com/article/2175826/software/linus-torvalds-suspends-key-linux-developer.html">issues</a> and <a href="http://boycottsystemd.org/">controversies</a> involved. You probably also have also heard about some <a href="http://uselessd.darknedgy.net/">new</a> <a href="http://en.wikipedia.org/wiki/OpenRC">alternatives</a>.<br />
<br />
Now instead of boring you with long lengthy arguments and debatable concepts and anecdotes, I'm going to boil the core problem with systemd down to two simple points:<br />
<br />
<ol>
</ol>
<ul>
<li><span style="font-size: large;">The singular aim of systemd is to get other projects to depend on it.</span></li>
<li><span style="font-size: large;">The systemd developers have a track record for <b>not</b> knowing what they're doing, thereby creating very fragile and unwieldy problematic software.</span></li>
</ul>
<ol>
</ol>
Now if you're looking for proof of these two points, you're free to read through the aforementioned pages I linked to, as well as search online for more. I hope you've enjoyed my summary.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com84tag:blogger.com,1999:blog-833174317742362874.post-90269871314685087842014-06-30T20:24:00.000-07:002014-07-01T07:41:51.433-07:00Memory management in C and auto allocating sprintf() - asprintf()<h3 style="text-align: center;">
Memory Management</h3>
Memory management in C is viewed by some to be quite tricky. One needs to work with pointers that can point anywhere in memory, and if misused, cause a program to misbehave, or worse.<br />
<br />
The basic functions to allocate and deallocate memory in C are <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/malloc.html">malloc()</a></i> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/free.html"><i>free()</i></a> respectively. The <i>malloc()</i> function takes a size in bytes of how much to allocate, and returns a pointer to the allocated memory to be used. Upon failure, a <a href="http://en.wikipedia.org/wiki/Pointer_%28computer_programming%29#Null_pointer">null pointer</a>, which can be thought of as a pointer pointing to <i>0</i> is returned. The <i>free()</i> function takes the pointer returned by <i>malloc()</i>, and deallocates the memory, or frees up the memory it was once pointing to.<br />
<br />
To work with malloc() in simple situations, typically, code along the following lines is used:<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">void *p = malloc(size);<br />if (p)<br />{<br /> ... work with p ...<br /> free(p);<br />}<br />else<br />{<br /> ... handle error scenario ...<br />}</span></blockquote>
<br />
Unfortunately many experienced programmers forget to handle the failure scenario. I've even heard some say they purposely don't, as they have no clue how to proceed, and just letting the program crash is good enough for them. If you meet someone who makes that argument, revoke their programming license. We don't need such near sighted idiots writing libraries.<br />
<br />
In any case, even the above can lead to some misuse. After this block of code runs, what is <i>p</i> now pointing to?<br />
<br />
After the above code runs, in the case that <i>malloc()</i> succeeded, <i>p</i> is now pointing to memory in middle of nowhere, and can't be used. This is known as a <a href="http://en.wikipedia.org/wiki/Dangling_pointer">dangling pointer</a>. Dangling pointers can be dangerous, as an <i>if</i> clause as above will think the pointer is valid, and may do something with it, or lead to the infamous <i>use after free</i> bug. This becomes more likely to occur as the situation becomes more complicated and there are loops involved, and how <i>malloc()</i> and <i>free()</i> interact can take multiple paths.<br />
<br />
Pointers involved with memory management should always be pointing at <i>0</i> or at allocated memory. Anything else is just asking for trouble. Therefore, I deem any direct use of <i>free()</i> dangerous, as it doesn't set the pointer to <i>0</i>.<br />
<br />
So if <i>free()</i> is considered harmful, what should one use?<br />
<br />
In C++, I recommend the following:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">static inline void insane_free(void *&p)<br />{<br /> free(p);<br /> p = 0;<br />}</span></blockquote>
<br />
This <i>insane_free()</i> is now a drop in replacement for <i>free()</i>, and can be used instead. (Since C++ programs normally use <i>new</i> and <i>delete</i>/<i>delete[]</i> instead, I leave it as an exercise to the reader how to work with those.)<br />
<br />
However, C doesn't support direct references. One can pass a pointer by a pointer to accomplish similar results, but that becomes clunky and is not a drop in replacement. So in C, I recommend the following:<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">#define insane_free(p) { free(p); p = 0; }</span></blockquote>
It makes use of the preprocessor, so some may consider it messy, but it can be used wherever <i>free()</i> currently is. One could also name the macro <i>free</i> in order to automatically replace existing code, but it's best not to program that way, as you begin to rely on these semantics. This in turn means someone copying your code may think a call to <i>free()</i> is the normal <i>free()</i> and not realize something special is occurring when they <a href="http://insanecoding.blogspot.com/2014/05/copying-code-copying-implementation.html">copy it elsewhere without the macro</a>.<br />
<br />
Correct usage in simple cases is then:<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">void *p = malloc(size);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">if (p)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> ... work with p ...</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> insane_free(p);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">else</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> ... handle error scenario ...</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span></blockquote>
</blockquote>
If you think using a wrapper macro or function is overkill, and just always manually assigning the pointer to <i>0</i> after freeing is the way to go, consider that it's unwieldy to constantly do so, and you may forget to. If the above technique was always used, all <i>use after free</i> bugs would never have occurred in the first place.<br />
<br />
Something else to be aware of is that there's nothing wrong with calling <i>free(0)</i>. However, calling <i>free()</i> upon a pointer which is not null and not pointing to allocated memory is forbidden and will crash your program. So stick to the advice here, and you may just find memory management became significantly easier.<br />
<br />
If all this talk of pointers is beyond you, consider acquiring <a href="http://www.amazon.com/gp/product/1449344186/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1449344186&linkCode=as2&tag=insacodi-20&linkId=WE4BS3IXKBI3ENS5">Understanding and Using C Pointers</a>.<br />
<br />
<h3 style="text-align: center;">
<a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/sprintf.html"><i>sprintf()</i></a> and <a href="http://asprintf.insanecoding.org/"><i>asprintf()</i></a></h3>
If you do a lot of C programming, at some point, you probably have used the <i>sprintf()</i> function, or its safer counterpart <i>snprintf()</i>.<br />
<br />
These two functions <i>sprintf()</i> and <i>snprintf()</i> act like <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/printf.html">printf()</a></i>, but instead of printing to the <a href="http://en.wikipedia.org/wiki/Standard_streams#Standard_output_.28stdout.29">standard output</a>, they print to a fixed-length string buffer. Now a fixed-length string buffer is great and all, but what if you wanted something which automatically allocated the amount it needed?<br />
<br />
Enter <i><a href="http://asprintf.insanecoding.org/">asprintf()</a></i>, a function which acts like <i>sprintf()</i>, but is auto-allocating, and needs no buffer supplied. This function was invented ages ago by <a href="http://en.wikipedia.org/wiki/GNU_C_Library">GLIBC</a>, shortly thereafter copied to the modern <a href="http://en.wikipedia.org/wiki/Berkeley_Software_Distribution">BSDs</a>, and found its way further still into all sorts of libraries, although is not yet ubiquitous.<br />
<br />
Let's compare the prototypes of the two:<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><i>int</i> sprintf(<i>char *</i>buffer, <i>const char *</i>format, ...);</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><span style="font-style: italic;">int</span> asprintf(<span style="font-style: italic;">char **</span>ret, <span style="font-style: italic;">const char *</span>format, <span style="font-style: italic;">...</span>);</span></blockquote>
The sanest approach would have been for a function like asprintf() to have the prototype of:<br />
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace; font-style: italic;">char *</span><span style="font-family: 'Courier New', Courier, monospace;">asprintf(c</span><span style="font-family: 'Courier New', Courier, monospace; font-style: italic;">onst char *</span><span style="font-family: 'Courier New', Courier, monospace;">format, </span><span style="font-family: 'Courier New', Courier, monospace; font-style: italic;">...</span><span style="font-family: 'Courier New', Courier, monospace;">);</span></blockquote>
But its creators wanted to make it act like sprintf(), and its design can also be potentially more useful.<br />
<br />
Instead of passing <i>asprintf()</i> a buffer, a pointer to a variable of type <i>char *</i> needs to be passed, like so:<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">char *buffer;</span><br />
<span style="font-family: Courier New, Courier, monospace;">asprintf(&buffer, ...whatever...);</span></blockquote>
</blockquote>
Now how asprintf() actually works is no big secret. The <a href="http://en.wikipedia.org/wiki/C99">C99</a> standard specified that snprintf() upon failure should return the amount of characters that would be needed to contain its output. Which means that conceptually something along the following lines would be all that asprintf() needs to do:<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">char *buffer = malloc(snprintf(0, 0, format, data...)+1);</span><br />
<span style="font-family: Courier New, Courier, monospace;">sprintf(buffer, format, data...);</span></blockquote>
</blockquote>
Of course though, the above taken verbatim would be incorrect, because it mistakenly assumes that nothing can go wrong, such as the <i>malloc()</i> or <i>snprintf()</i> failing.<br />
<br />
First let's better understand what the <i>*printf()</i> functions return. Upon success, they return the amount of characters written to the string (which does not include the trailing <a href="http://en.wikipedia.org/wiki/Null_character">null byte</a>). Or in other words, the return value is equivalent to calling <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strlen.html">strlen()</a></i> on the data being output, which can save you needing to use a <i>strlen()</i> call with <i>sprintf()</i> or similar functions for certain scenarios. Upon failure, for whatever reason, the return is <i>-1</i>. Of course there's the above mentioned exception to this with <i>snprintf()</i>, where the amount of characters needed to contain the output would be returned instead. If during the output, the size overflows (exceeds <i><a href="http://www.cplusplus.com/reference/climits/">INT_MAX</a></i>), many implementations will return a large negative value (failure with <i>snprintf(), or success with all the functions</i>).<br />
<br />
Like the other functions, <i>asprintf()</i> also returns an integer of the nature described above. Which means working with <i>asprintf()</i> should go something like this:<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">char *buffer;</span><br />
<span style="font-family: Courier New, Courier, monospace;">if (asprintf(&buffer, ...whatever...) != -1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> do_whatever(buffer);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> insane_free(buffer);</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span></blockquote>
</blockquote>
However, unlike the other functions, <i>asprintf()</i> has a second return value, its first argument, or what the function sees as <i>*ret</i>. To comply with the memory management discussion above, this should also be set to <i>0</i> upon failure. Unfortunately, many popular implementations, including those in GLIBC and <a href="http://en.wikipedia.org/wiki/MinGW">MinGW</a> fail to do so.<br />
<br />
Since I develop with the above systems, and I'm using <i>asprintf()</i> in loops with multiple paths, it becomes unwieldy to need to pass around the buffer and the returned <i>int</i>, so I'd of course want saner semantics which don't leave dangling pointers in my program.<br />
<br />
In order to correct such mistakes, I would need to take code from elsewhere, or whip up my own function. Now I find developing functions such as these to be relatively simple, but even so, I always go to check other implementations to see if there's any important points I'm missing before I go implement one. Maybe, I'll even find one which meets my standards with a decent license which I can just copy verbatim.<br />
<br />
In researching this, to my shock and horror, I came across implementations which properly ensure that <i>*ret</i> is set to <i>0</i> upon failure, but the returned <i>int</i> may be undefined in certain cases. That some of the most popular implementations get one half wrong, and that some of the less popular get the other half wrong is just downright terrifying. This means that there isn't any necessarily portable way to check for failure with the different implementations. I certainly was not expecting that, but with the amount of horrible code out there, I guess I really shouldn't be surprised anymore.<br />
<br />
Also in the course of research, besides finding many implementations taking a non-portable approach, many have problems in all sorts of edge cases. Such as mishandling overflow, or not realizing that two successive calls to a <i>*printf()</i> function with the same data may not necessarily yield the same results. Some try to calculate the length needed with some extra logic and only call <i>sprintf()</i> once, but this logic may not be portable, or always needs updating as new types are added to the format string as standards progress, or the C library decided to offer new features. Some of the mistakes I found seem to be due to expecting a certain implementation of underlying functions, and then later the underlying functions were altered, or the <a href="http://insanecoding.blogspot.com/2014/05/copying-code-copying-implementation.html">code was copied verbatim to another library, without noting the underlying functions acted differently</a>.<br />
<br />
So, once again, I'm finding myself needing to supply the world with more <a href="http://asprintf.insanecoding.org/"><b><i>usable implementations</i></b></a>.<br />
<br />
Let's dive into how to implement <i>asprinf()</i>.<br />
<br />
Every one of these kind of functions actually has two variants, the regular which takes an unlimited amount of arguments, and the <i>v</i> variants which take a <a href="http://en.cppreference.com/w/c/variadic/va_list"><i>va_list</i></a> (defined in <a href="http://en.wikipedia.org/wiki/Stdarg.h"><i>stdarg.h</i></a>) after the format argument instead. These <i>va_list</i>s are what <i>...</i> gets turned into after use, and in fact, every non-<i>v</i> <u><i>*printf()</i></u> function is actually wrapped to a counterpart <i>v*printf()</i> function. This makes implementing <i>asprintf()</i> itself quite straight forward:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-bEUMLltnnYc/U7IuU4otihI/AAAAAAAAAMQ/1dVl_TTkdi4/s1600/asprintf.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-bEUMLltnnYc/U7IuU4otihI/AAAAAAAAAMQ/1dVl_TTkdi4/s1600/asprintf.png" height="116" width="320" /></a></div>
<br />
<br />
<div>
To fix the aforementioned return problems, one could also easily throw in here a check upon the correct return variable used in the underlying <i>vasprintf()</i> implementation and use it to set the other. However, that's not a very portable fix, and the underlying implementation of <i>vasprintf()</i> can have other issues as described above.</div>
<div>
<br /></div>
<div>
A straight forward implementation of <i>vasprintf()</i> would be:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-zhik6v2VyVA/U7IuciOO_HI/AAAAAAAAAMY/hMqppCIvzD0/s1600/vasprintf-c99.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-zhik6v2VyVA/U7IuciOO_HI/AAAAAAAAAMY/hMqppCIvzD0/s1600/vasprintf-c99.png" height="276" width="320" /></a></div>
<br />
<br /></div>
<div>
As long as you have a proper C99 implementation of <i>stdarg.h</i> and <i>vsnprintf()</i>, you should be good to go. However, some systems may have <i>vsnprintf()</i> but not <i>va_copy()</i>. The <i>va_copy()</i> macro is needed because a <i>va_list</i> may not just be a simple object, but have handles to elsewhere, and a <i><a href="http://en.wikipedia.org/wiki/Object_copy#Deep_copy">deep copy</a></i> is needed. Since <i>vsnprintf()</i> being passed the original <i>va_list</i> may modify its contents, a copy is needed because the function is called twice.</div>
<div>
<br /></div>
<div>
Microsoft Visual C++ (MSVC, or Microsoft Vs. C++ as I like to think of it) up until the latest versions has utterly lacked <i>va_copy()</i>. This and several other systems that lack it though usually have simple <i>va_list</i>s that can be <i><a href="http://en.wikipedia.org/wiki/Object_copy#Shallow_copy">shallow copied</a></i>. To gain compatibility with them, simply employ:<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<br />
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">#ifndef va_copy</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">#define va_copy(dest, src) dest = src</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">#endif</span></blockquote>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
Be warned though that if your system lacks <i>va_copy()</i>, and a <i>deep copy</i> is required, using the above is a recipe for disaster.</div>
<div>
<br /></div>
<div>
Once we're dealing with systems where shallow copy works though, the following works just as well, as <i>vsnprintf()</i> will be copying the <i>va_list</i> it receives and won't be modifying other data.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-BGeRD3fMzko/U7IuzcphwWI/AAAAAAAAAMg/tcm2FDLPZQw/s1600/vasprintf-novacopy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-BGeRD3fMzko/U7IuzcphwWI/AAAAAAAAAMg/tcm2FDLPZQw/s1600/vasprintf-novacopy.png" height="218" width="320" /></a></div>
<br />
<br /></div>
<div>
Before we go further, there's two points I'd like to make.</div>
<div>
<ul>
<li>Some implementations of <i>vsnprintf()</i> are wrong, and always return <i>-1</i> upon failure, not the size that would've been needed. On such systems, another approach will need to be taken to calculate the length required, and the implementations here of <i>vasprintf()</i> (and by extension <i>asprintf()</i>) will just always return <i>-1</i> and <i>*ret</i> (or <i>*strp</i>) will be <i>0</i>.</li>
<li>The code <span style="font-family: 'Courier New', Courier, monospace;">if ((r < 0) || (r > size))</span> could instead be <span style="font-family: Courier New, Courier, monospace;">if (r != size)</span>, more on that later.</li>
</ul>
<div>
Now on Windows, <i>vsnprintf()</i> always returns -1 upon failure, in violation of the C99 standard. However, in violation of <a href="http://msdn.microsoft.com/en-us/library/1kt27hek.aspx">Microsoft's own specifications</a>, and undocumented, I found that <i>vsnprintf()</i> with the first two parameters being passed <i>0</i> as in the above code actually works correctly. It's only when you're passing data there that the Windows implementation violates the spec. But in any case, <a href="http://insanecoding.blogspot.com/2010/06/undocumented-apis.html">relying on undocumented behavior is never a good idea</a>.</div>
</div>
<div>
<br /></div>
<div>
On certain versions of MinGW, if <i>__USE_MINGW_ANSI_STDIO</i> is defined before <a href="http://en.wikipedia.org/wiki/C_file_input/output"><i>stdio.h</i></a> <stdio .h="">is included, it'll cause the broken Windows <i>*printf()</i> functions to be replaced with C99 standards compliant ones.</stdio></div>
<div>
<br /></div>
<div>
In any case though, Windows actually provides a different function to retrieve the needed length, <i><a href="http://msdn.microsoft.com/en-us/library/w05tbk72.aspx">_vscprintf()</a></i>. A simple implementation using it would be:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-48uFXDHU9jg/U7Iu7k8yBbI/AAAAAAAAAMo/XmwVVhdWfjE/s1600/vasprintf-msvc-nofuture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-48uFXDHU9jg/U7Iu7k8yBbI/AAAAAAAAAMo/XmwVVhdWfjE/s1600/vasprintf-msvc-nofuture.png" height="162" width="320" /></a></div>
<br />
<br /></div>
<div>
This however makes the mistake of assuming that<i> vsnprintf()</i> is implemented incorrectly as it currently is with MSVC. Meaning this will break if Microsoft ever fixes the function, or you're using MinGW with <i>__USE_MINGW_ANSI_STDIO</i>. So better to use:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-3cNvg0nBEd0/U7IvBiIpvkI/AAAAAAAAAMw/n9yByNICpPs/s1600/vasprintf-msvc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-3cNvg0nBEd0/U7IvBiIpvkI/AAAAAAAAAMw/n9yByNICpPs/s1600/vasprintf-msvc.png" height="213" width="320" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
Lastly, let me return to that second point from earlier. The <i>vsnprintf()</i> function call the second time may fail because the system ran out of memory to perform its activities once the call to <i>malloc()</i> succeeds, or something else happens on the system to cause it to fail. But also, in a <i><a href="http://en.wikipedia.org/wiki/Multithreading_%28software%29#Multithreading">multi-threaded</a></i> program, the various arguments being passed could have their data change between the two calls.</div>
<div>
<br /></div>
<div>
Now if you're calling functions while another thread is modifying the same variables you're passing to said function, you're just asking for trouble. Personally, I think that all the final check should do is ensure that <i>r</i> is equal to <i>size</i>, and if not, something went wrong, free the data (with <i>insane_free()</i> of course), and set <i>r</i> to <i>-1</i>. However, any value between <i>0</i> and <i>size</i> (inclusive), even when not equal to <i>size</i> means the call succeeded for some definition of success, which the above implementations all allow for (except where not possible Microsoft). Based on this idea, several of the implementations I looked at constantly loop while <i>vsnprintf()</i> continues to indicate that the buffer needs to be larger. Therefore, I'll provide such an implementation as well:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-veeaAaPeZu4/U7IvHB4zWcI/AAAAAAAAAM4/CEHNhzu16Zc/s1600/vasprintf-c99-loop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-veeaAaPeZu4/U7IvHB4zWcI/AAAAAAAAAM4/CEHNhzu16Zc/s1600/vasprintf-c99-loop.png" height="230" width="320" /></a></div>
<br />
<br /></div>
<div>
Like the first implementation, if all you lacked was <i>va_copy()</i>, and <i>shallow copy</i> is fine, it's easy to get this to work on your platform as described above. But if <i>vsnprintf()</i> isn't implemented correctly (hello MSVC), this will always fail.</div>
<div>
<br /></div>
<div>
All the code here including the appropriate headers, along with notes and usage examples are all packed up and ready to use on my <a href="http://asprintf.insanecoding.org/"><i>asprintf()</i> implementation</a> website. Between everything offered, you should hopefully find something that works well for you, and is better than what your platform provides, or alternative junk out there.</div>
<div>
<br /></div>
<div>
As always, I'm only human, so if you found any bugs, please inform me.</div>
insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com289tag:blogger.com,1999:blog-833174317742362874.post-5366867982240175732014-06-22T22:21:00.000-07:002014-06-22T22:44:24.515-07:00Avoid incorrect ChaCha20 implementations<a href="http://cr.yp.to/chacha.html">ChaCha20</a> is a <a href="http://en.wikipedia.org/wiki/Stream_cipher">stream cipher</a> which is gaining a lot of popularity of late. Practically every library today which provides ciphers seems to have it as an addition in their latest releases.<br />
<br />
In cryptography, there are two kinds of ciphers, <a href="http://en.wikipedia.org/wiki/Block_cipher">block ciphers</a> and stream ciphers. Block ciphers are where the underlying algorithm works with data with a certain fixed chunk size (or block). Popular blocks sizes are 16 and 64 bytes. Stream ciphers are effectively block ciphers where the chunk size is a single byte.<br />
<br />
Classical stream ciphers, such as <a href="http://en.wikipedia.org/wiki/RC4">RC4</a>, can work with data of arbitrary size, although every single byte is dependent on every previous byte. Which means encryption/decryption cannot begin in the middle of some data, and maintain compatibility where some other starting point was used. Block ciphers generally can have their blocks encrypted and decrypted arbitrarily, with none dependent upon any other, however, they cannot work with data of arbitrary size.<br />
<br />
In order to allow block ciphers to work with data of arbitrary size, one needs to <a href="http://en.wikipedia.org/wiki/Padding_%28cryptography%29#Block_cipher_mode_of_operation">pad the data</a> to be encrypted to a multiple of the block size. However, a clever alternative is <a href="http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29">counter mode</a>.<br />
<br />
Different modes for working with block ciphers exist. Some try to improve security by making each block depend on every other, some utilize various interesting techniques for other properties. Counter mode does not encrypt the desired data (the <i>plaintext)</i> directly, rather, an ever incrementing counter is encrypted. The result of this encryption is then <a href="http://en.wikipedia.org/wiki/XOR_cipher">xored</a> with the desired data.<br />
<br />
Counter mode effectively turns a block cipher into a stream cipher, as the <i>plaintext</i> is never actually passed to the block cipher. Rather, a counter which is a multiple of the block size is used. One can always xor bytes with an arbitrary size, and since that is the only step in counter mode against the plain text, it is effectively a stream cipher. Since the underlying cipher can be a block cipher with no dependency between blocks, this kind of stream cipher also allows one to jump ahead to any particular multiple of the block size in the data, and begin encryption/decryption from there.<br />
<br />
Now while ChaCha20 is advertised as a stream cipher, it's actually designed as a block cipher in counter mode. The internal design mostly mirrors that of typical counter mode design, except that the counter components are directly fused with a large but simple block cipher. Since it's really a block cipher, it has an internal block size, and also allows one to jump ahead to some multiple of it.<br />
<br />
Since ChaCha20 is considered to have a great level of security, and all these other wonderful properties, it's starting to see a lot of use. However, practically every implementation I'm seeing is either utterly broken, or has some ridiculous API.<br />
<br />
Common ChaCha20 implementation mistakes:<br />
<ul>
<li>Implemented as a typical block cipher, not allowing usage with arbitrary amounts of bytes, or worse, the API allows for it, but produces incorrect results.</li>
<li>Implemented as a typical stream cipher with no way to jump ahead.</li>
<li>Failing on <a href="http://en.wikipedia.org/wiki/Endianness">Big-Endian</a> systems.</li>
</ul>
<br />
The first mistake I listed is the most common. If some software is only using ChaCha20 internally, and always using it in a multiple of its block size (or it's all the crummy API offers), then things are fine. But if it's a library which is inviting others to use it, and it can be used incorrectly, expect disaster to ensue.<br />
<br />
The reference implementation of ChaCha20 was designed that an arbitrary amount of data can be encrypted, as long as all but the last usage of the API was a multiple of the block size. This was also mentioned in its documentation. However, practically every other implementation out there copies this design in some way, but makes no note of it. Worse yet, some libraries are offering ChaCha20 with this implementation flaw alongside other stream ciphers with an identical API whereas those can be used arbitrarily throughout.<br />
<br />
Essentially, this means if you're using ChaCha20 right now in a continuous fashion with chunks of various sizes, your data is being encrypted incorrectly, and won't be interoperable with other implementations. These broken implementations are able to output exactly one chunk correctly which is not a multiple of the block size, which destroys their internal buffers, and screws up every output thereafter.<br />
<br />
I noticed a <a href="http://insanecoding.blogspot.com/2007/05/hashing-out-hashing-and-hash-libraries.html">similar situation</a> with hash algorithm implementations several years back. However, most hash implementations are fine. Yet with ChaCha20, practically every implementation I looked at recently was broken.<br />
<br />
Since this situation cannot stand, especially with ChaCha20 gaining speed, I am providing a <i><b><a href="http://chacha20.insanecoding.org/">simple implementation</a></b></i> without these flaws. This implementation is designed to be correct, portable, and simple. (Those wanting an optimized version of this should consider paying for more optimized routines)<br />
<br />
Usage of the <a href="http://en.wikipedia.org/wiki/C99">C99</a> API I designed is as follows:<br />
<br />
Custom type: <i style="font-family: "Courier New",Courier,monospace;">chacha20_ctx</i><br />
This type is used as a context for a state of encryption.<br />
<br />
To initialize:<br />
<div style="font-family: "Courier New",Courier,monospace;">
<i>void</i> chacha20_setup(<i>chacha20_ctx *</i>ctx, <i>const uint8_t *</i>key, <i>size_t</i> length, <i>uint8_t</i> nonce[8]);</div>
<br />
The encryption key is passed via a pointer to a byte array and its length in bytes. The key can be 16 or 32 bytes. The <a href="http://en.wikipedia.org/wiki/Cryptographic_nonce">nonce</a> is always 8 bytes.<br />
<br />
Once initialized, to encrypt data:<br />
<div style="font-family: "Courier New",Courier,monospace;">
<i>void</i> chacha20_encrypt(<i>chacha20_ctx *</i>ctx, <i>const uint8_t *</i>in, <i>uint8_t *</i>out, <i>size_t</i> length);</div>
<br />
You can pass an arbitrary amount of data to be encrypted, just ensure the output buffer is always at least as large as the input buffer. This function can be called repeatedly, and it doesn't matter what was done with it previously.<br />
<br />
To decrypt data, initialize, and then call the decryption function:<br />
<div style="font-family: "Courier New",Courier,monospace;">
<i>void</i> chacha20_decrypt(<i>chacha20_ctx *</i>ctx, <i>const uint8_t *</i>in, <i>uint8_t *</i>out, <i>size_t</i> length);</div>
<br />
For encryption or decryption, if you want to jump ahead to a particular block:<br />
<div style="font-family: "Courier New",Courier,monospace;">
<i>void</i> chacha20_counter_set(<i>chacha20_ctx *</i>ctx, <i>uint64_t</i> counter);</div>
<br />
Counter is essentially the number of the next block to encrypt/decrypt. ChaCha20's internal block size is 64 bytes, so to calculate how many bytes are skipped by a particular counter value, multiply it by 64.<br />
<br />
In addition to just providing a library, I gathered the test vectors that were submitted for various <a href="http://en.wikipedia.org/wiki/Request_for_Comments">RFCs</a>, and included a series of unit tests to test it for correctness.<br />
<br />
For fun, since I'm also playing around a bit with <a href="http://insanecoding.blogspot.com/2014/04/libressl-good-and-bad.html">LibreSSL</a> these days, I wrapped its API up in the API I described above. The wrapper is included in my package with the rest of the code, however it is currently not designed for serious usage outside of the included test cases.<br />
<br />
Since I already whipped up some unit tests that anyone can use, I'll leave it as an exercise to the reader to determine which libraries are and aren't implemented correctly.<br />
<br />
I tried to ensure my library is bug free, but I am only human. If you find a mistake, please report it.<br />
<br />insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com55tag:blogger.com,1999:blog-833174317742362874.post-29758524676484846282014-05-21T16:00:00.000-07:002014-05-21T16:13:15.810-07:00LibreSSL porting updateI've recently covered some issues with <a href="http://insanecoding.blogspot.com/2014/04/libressl-good-and-bad.html">LibreSSL</a> and some <a href="http://insanecoding.blogspot.com/2014/04/common-libressl-porting-mistakes.html">common porting mistakes</a>.<br />
<br />
Since these articles came out, I've noticed two broken ports I saw prior seem to have vanished. One port has seen significant improvement in response to these articles, although still has significant concerns. And worst of all, more ports are popping up.<br />
<br />
The official team has since <a href="http://www.openbsd.org/papers/bsdcan14-libressl/mgp00030.html">reiterated</a> some of these concerns, and I also wrote <a href="http://insanecoding.blogspot.com/2014/05/a-good-idea-with-bad-usage-devurandom.html">two</a> <a href="http://insanecoding.blogspot.com/2014/05/dealing-with-randomness.html">articles</a> regarding some concerns with random data.<br />
<br />
Unfortunately, many of these ports are continuing to rely on <i>arc4random()</i> implementations on certain OSs or from certain portability libraries. These OSs or libraries may be copying some or all of the code from OpenBSD, but they are <a href="http://insanecoding.blogspot.com/2014/05/copying-code-copying-implementation.html">not copying the implementation</a>.<br />
<br />
To demonstrate this, let's see how different implementations of <i>arc4random()</i> work across <i>fork()</i> using the following test code:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: grey; font-style: italic;">/*</span>
<span style="color: grey; font-style: italic;">Blogger is refusing to allow me to list the headers without trying to escape the signs.</span>
<span style="color: grey; font-style: italic;">So they are: stdio.h, stdlib.h, stdint.h, unistd.h, sys/wait.h</span>
<span style="color: grey; font-style: italic;">And on Linux: bsd/stdlib.h</span>
<span style="color: grey; font-style: italic;">*/</span>
<span style="color: maroon;">int</span><span style="color: black;"> main()</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> children = </span><span style="color: blue;">3</span><span style="color: black;">;</span>
<span style="color: black;"> pid_t pid = getpid();</span>
<span style="color: black;"> printf(</span><span style="color: #dd0000;">"parent process %08x: %08x %08x</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, (</span><span style="color: maroon;">uint32_t</span><span style="color: black;">)pid, arc4random(), arc4random());</span>
<span style="color: black;"> fflush(stdout);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">while</span><span style="color: black;"> (children--)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> pid_t pid = fork();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (pid > </span><span style="color: blue;">0</span><span style="color: black;">) </span><span style="color: grey; font-style: italic;">//Parent</span>
<span style="color: black;"> {</span>
<span style="color: black;"> waitpid(pid, </span><span style="color: blue;">0</span><span style="color: black;">, </span><span style="color: blue;">0</span><span style="color: black;">);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (pid == </span><span style="color: blue;">0</span><span style="color: black;">) </span><span style="color: grey; font-style: italic;">//Child</span>
<span style="color: black;"> {</span>
<span style="color: black;"> pid = getpid();</span>
<span style="color: black;"> printf(</span><span style="color: #dd0000;">" child process %08x: %08x %08x</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, (</span><span style="color: maroon;">uint32_t</span><span style="color: black;">)pid, arc4random(), arc4random());</span>
<span style="color: black;"> fflush(stdout);</span>
<span style="color: black;"> _exit(</span><span style="color: blue;">0</span><span style="color: black;">);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Error</span>
<span style="color: black;"> {</span>
<span style="color: black;"> perror(</span><span style="color: blue;">0</span><span style="color: black;">);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">break</span><span style="color: black;">;</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> printf(</span><span style="color: #dd0000;">"parent process %08x: %08x %08x</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, (</span><span style="color: maroon;">uint32_t</span><span style="color: black;">)pid, arc4random(), arc4random());</span>
<span style="color: black;"> fflush(stdout);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(</span><span style="color: blue;">0</span><span style="color: black;">);</span>
<span style="color: black;">}</span>
</pre>
<br />
<br />
OpenBSD (the reference implementation):<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000660d: beb04672 aa183dd0</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 00001a2a: e52e0b25 764966bb</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 00007eb7: 27619dd1 a7c0df81</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 000039f5: 33daf1f1 4524c6c6</span><br />
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000660d: 1eb05b45 d3956c43</span></blockquote>
Linux with libbsd 0.6:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">parent process 000031cb: 2bcaaa9a 01532d3f</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 000031cc: 3b43383f 4fbbb4d5</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 000031cd: 3b43383f 4fbbb4d5</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 000031ce: 3b43383f 4fbbb4d5</span><br />
<span style="font-family: "Courier New",Courier,monospace;">parent process 000031cb: 3b43383f 4fbbb4d5</span></blockquote>
NetBSD 6.1.2:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000021a: 4bc81424 958bf90f</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 0000021f: c0681a36 5a3f8bdb</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 00000022: c0681a36 5a3f8bdb</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 000001fc: c0681a36 5a3f8bdb</span><br />
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000021a: c0681a36 5a3f8bdb</span></blockquote>
FreeBSD 9.2:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000032e: 03d19ad2 543c5fa4</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 0000032f: 6e3a1214 57b74381</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 00000330: 6e3a1214 57b74381</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> child process 00000331: 6e3a1214 57b74381</span><br />
<span style="font-family: "Courier New",Courier,monospace;">parent process 0000032e: 6e3a1214 57b74381</span></blockquote>
DragonFlyBSD 3.4.3:<br />
<blockquote class="tr_bq">
<div style="font-family: "Courier New",Courier,monospace;">
parent process 0000030a: cb987922 8f94fb58<br />
child process 0000030b: 65047965 1ebdc52b<br />
child process 0000030c: 65047965 1ebdc52b<br />
child process 0000030d: 65047965 1ebdc52b<br />
parent process 0000030a: 65047965 1ebdc52b</div>
</blockquote>
<br />
So in looking at this data, one can see that on OpenBSD the random data is utterly different between the parent and all the various children. However, in all the ports of the function, the parent and children all share the exact same state after the <i>fork()</i> call. This situation is fine for single-process programs, but is a disaster in multi-process ones.<br />
<br />
Since LibreSSL is having its random needs all being supplied by <i>arc4random*()</i>, and it can be used by multi-process servers, there is a serious porting problem here.<br />
<br />
I covered this problem without elaboration in my <a href="http://insanecoding.blogspot.com/2014/05/dealing-with-randomness.html">previous article</a>. See there for some solutions. The OpenBSD team is saying randomness is the responsibility of the OS, and for many of the issues involved, they are quite right. However, remember your ports cannot necessarily rely on the random functions provided by your OS. Even if the OSs fix them in future versions, one still has to be mindful of porting to older ones, so ensure your port doesn't rely too much on the OS.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com29tag:blogger.com,1999:blog-833174317742362874.post-38080757099314474102014-05-20T12:01:00.000-07:002014-05-31T11:19:41.004-07:00Dealing with randomnessTwo weeks ago, I wrote an <a href="http://insanecoding.blogspot.com/2014/05/a-good-idea-with-bad-usage-devurandom.html">article</a> regarding randomness on UNIX systems and libraries. In it, I dealt with some theoretical API issues, and real world issues of libraries being horribly misdesigned. Today I'd like to focus more on the current state of things, and further discuss real world problems.<br />
<h4 style="text-align: center;">
Randomness</h4>
To start, let us understand what randomness means. <i>Any perceived randomness on your part is your inability to track all the variables. </i>Meaning that so called randomness to us mere mortals is something that we do not know nor can predict, at least not with the tools we have at our disposal.<br />
<br />
The ideas behind randomness and determinism have long been battled over by philosophers and scientists, with the former debating whether it's possible for such things to exist, and the latter hunting for its existence. But thankfully, in terms of cryptography, even though randomness doesn't exist, we can make believe it does, and we can generate data that appears random to anyone from the outside. We can do so by using variables that are hard to track and influence, and extremely confusing algorithms.<br />
<br />
Collecting unpredictable values is a challenge, and nothing is ever as unpredictable as one might think. Even if we were to look to quantum mechanics, which today is believed to be the closest thing to random that we know of, and measure some quantum behavior, the data might still be predicable, and therefore not random enough for cryptography needs. <a href="http://www.amazon.com/gp/product/0470474246/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470474246&linkCode=as2&tag=insacodi-20">Cryptography Engineering</a> pages 138-139 covers this in more detail, without even getting into possible advancements in the field. All the more so, being completely unpredictable is quite challenging using more conventional means.<br />
<br />
Essentially, we're left with trying to do our best, without really being able to ensure we're doing things well. The less information we leak, and more unpredictable we behave, the better.<br />
<h4 style="text-align: center;">
<b>Expectations</b></h4>
<br />
We tend to make assumptions. We make assumptions that the people behind Linux know how to create a great operating system. We assumed the OpenSSL team knew how to create a secure library. We assume the OpenBSD developers know safe software engineering. However, assumptions may be wrong. <i>Never assume, research and test it.</i><br />
<br />
Like others, I tend to assume that different pieces of software were written correctly. A couple of years back, I was having trouble with a web server in conjunction with IPv6, which was built on top of a network server library. I assumed the library written by supposed experts with IPv6 knowledge knew what they were doing, and blamed the newness of IPv6, and assumed the issue was not with the library but with the network stacks in the OS.<br />
<br />
For unrelated reasons I decided to improve my network development knowledge and purchased <a href="http://www.amazon.com/gp/product/0131411551/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131411551&linkCode=as2&tag=insacodi-20&linkId=5GUCHT6RQTHWNXYN">Unix Network Programming 3rd Edition</a>, which is now over a decade old, yet includes instructions on properly working with IPv6. Turns out I was able to fix the bug in the networking library I was using by modifying a single line in it. After reading this book, I also realized why I was having some rare hard to reproduce bug in another application, and fixed the issue in two minutes.<br />
<br />
The aforementioned book is considered the bible in terms of sockets programming. It explains common gotchas, and shows how to build robust networking software which works properly. We would assume the developers of a major networking library would have read it, but experience shows otherwise.<br />
<br />
I hardly need to elaborate how this has applied elsewhere (hello OpenSSL). <br />
<h4 style="text-align: center;">
False messiahs </h4>
Most software is written by people who aren't properly trained for their positions. They either get thrust into it, or dabble in something for fun, and it turns into something popular which people all over the globe use regularly. Suddenly popularity becomes the measurement for gauging expertise. Some kid with a spare weekend becomes the new recognized authority in some field with no real experience. People who end up in this position tend to start doing some more research to appear knowledgeable, and receive unsolicited advice, which to some extent propels them to be an expert in their field, at least at a superficial level. This can then further reinforce the idea that the previously written software was correct.<br />
<br />
Situations like this unfortunately even apply to security libraries and software which needs to be secure. We can read a book, which covers some important design nuance, and we'll think to ourselves that if we laymen read the popular book, surely those who need to be aware of its contents did, and assume the software we depend on every day is written properly. I've read a few books and papers over the years on proper random library gotchas, design, and more, yet used to simply use OpenSSL's functions in my code, as I assumed the OpenSSL developers read these books too and already took care of things for me.<br />
<br />
Recently, the OpenBSD team actually reviewed OpenSSL's random functions, and realized that no, the OpenSSL developers did not know what they were doing. Since the Debian key generation fiasco a couple of years ago revealed they were using uninitialized variables as part of randomness generation, I suspected they didn't know what they were doing, but never bothered looking any deeper. It's scary to realize you may in fact know how to handle things better than the so-called <i>experts</i>.<br />
<br />
I wrote an <a href="http://insanecoding.blogspot.com/2014/05/protecting-private-keys.html">article</a> the other day regarding how to protect private keys. A friend of mine after reading it asked me incredulously: <i>You mean Apache isn't doing this?!</i><br />
<br />
We have to stop blindly believing in certain pieces of software or developers. We must educate ourselves properly, and then ensure everything we use lives up to the best practices we know about.<br />
<h4 style="font-weight: normal; text-align: center;">
<b>UNIX random interfaces</b></h4>
Linux invented two interfaces for dealing with random data which were then copied by the other UNIX-like operating systems. <i>/dev/random</i> which produces something closely related to what the OS believes is random data it collected, and <i>/dev/urandom</i> which produces an endless stream of supposedly cryptographically secure randomish data. The difference in the output between the two of them should not be discernible, so theoretically, using one in place of the other should not make a difference.<br />
<br />
There's a ton of websites online which tell developers to use only /<i>dev/urandom</i>, since the whole point of a cryptographically-secure pseudo-random number generator is to appear random. So who needs <i>/dev/random</i> anyway, and finite amounts of randomishness is problematic, so everyone should just use something which is unlimited. Then the <a href="http://man7.org/linux/man-pages/man4/random.4.html">Linux manual page</a> will be blamed as a source of fostering confusion for suggesting there's situations that actually require <i>/dev/random</i>.<br />
<br />
Now those who have an ounce of critical thinking within themselves should be questioning, if there's never a point with <i>/dev/random</i>, why did Linux invent it in the first place? Why does it continue to provide it with the same semantics that it always had? In fact, the manual page is doing nothing more than paraphrasing and explaining the comments in the <a href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/char/random.c">source code</a>:<br />
<blockquote class="tr_bq">
<pre><code><span class="hl com"> * The two other interfaces are two character devices /dev/random and</span>
<span class="hl com"> * /dev/urandom. /dev/random is suitable for use when very high</span>
<span class="hl com"> * quality randomness is desired (for example, for key generation or</span>
<span class="hl com"> * one-time pads), as it will only return a maximum of the number of</span>
<span class="hl com"> * bits of randomness (as estimated by the random number generator)</span>
<span class="hl com"> * contained in the entropy pool.</span>
<span class="hl com"> *</span>
<span class="hl com"> * The /dev/urandom device does not have this limit, and will return</span>
<span class="hl com"> * as many bytes as are requested. As more and more random bytes are</span>
<span class="hl com"> * requested without giving time for the entropy pool to recharge,</span>
<span class="hl com"> * this will result in random numbers that are merely cryptographically</span>
<span class="hl com"> * strong. For many applications, however, this is acceptable. </span></code></pre>
</blockquote>
<pre><code><span class="hl com"> </span></code></pre>
There's actually a number of problems with pseudo-random number generators. They leak information about their internal states as they output their data. Remember, data is being generated from an internal state, there's an input which generates the stream of output, it's not just magically created out of thin air. The data being output will also eventually cycle. Using the same instance of a pseudo-random number generator over and over is a bad idea, as its output will become predictable. Not only will its future output become predictable, anything it once generated will also be known. Meaning pseudo-random number generators lack what is known as <a href="http://en.wikipedia.org/wiki/Forward_secrecy">forward secrecy</a>.<br />
<br />
Now if you believe the hype out there that for every situation, one should only use<i> /dev/urandom</i>, then you're implying you don't trust the developers of the Linux random interfaces to know what they're talking about. If they don't know what they're talking about, then clearly, they also don't know what they're doing. So why are you using any Linux supplied random interface, after all, Linux obviously handles randomness incorrectly!<br />
<br />
FreeBSD actually makes the above argument, as they only supply the <i>/dev/urandom</i> interface (which is known as <i>/dev/random</i> on FreeBSD), which uses random algorithms created by leading cryptographers, and nothing remotely like what Linux is doing. Each of the BSDs in fact claim that their solution to randomness is superior to all the other solutions found among competing operating systems.<br />
<br />
Linux on the other hand takes an approach where it doesn't trust the cryptographic algorithms out there. It has its own approach where it collects data, estimates the <a href="http://en.wikipedia.org/wiki/Entropy_%28computing%29">entropy</a> of that data, and uses its own modified algorithms for producing the randomness that it does. Cryptographers are constantly writing papers about how non-standard Linux is in what it does, and numerous suggestions are made to improve it, in one case, even a <a href="http://jlcooke.ca/random/">significant patch</a>.<br />
<br />
Now I personally dislike Linux's approach to throw cryptographic practice out the window, but on the other hand, I like their approach in not putting too much faith into various algorithms. I like FreeBSD's approach to using well designed algorithms by cryptographers, but I dislike their inability to ensure the highest levels of security for when you really need it.<br />
<br />
The cryptographers out there are actually divided on many of the issues. Some believe entropy estimation is utterly flawed, and cryptographically strong algorithms are all you need. Others believe that such algorithms are more easily attacked and prone to catastrophic results, and algorithms combined with pessimistic estimators and protective measures should be used.<br />
<br />
In any case, while the various operating systems may be basing themselves on various algorithms, are they actually implementing them properly? Are they being mindful of all the issues discussed in the various books? I reviewed a couple of them, and I'm not so sure.<br />
<br />
A strong point to consider for developers is that even though OS developers may improve their randomness in response to various papers those developers happen to stumble across or gets shoved in their faces, it doesn't necessarily mean you're free from worrying about the issues in your applications. It's common to see someone compiling and using some modern software package on say Red Hat Enterprise Linux 5, with its older version of Linux. I recently got a new router which allows one to <a href="http://en.wikipedia.org/wiki/Secure_Shell">SSH</a> into it. Doing so, I saw it had some recentish version of <a href="http://en.wikipedia.org/wiki/BusyBox">BusyBox</a> and some other tools on it, but was running on Linux 2.4!<br />
<h4 style="text-align: center;">
Situations to be mindful of</h4>
There are many situations one must be mindful of when providing an interface to get random data. This list can be informative, but is not exhaustive.<br />
<br />
Your system may be placed onto a router or similar device which generates private keys upon first use. These devices are manufactured in bulk, and are all identical. These devices also generally lack a hardware clock. In typical operating conditions, with no special activity occurring, these devices will be generating identical keys, there's nothing random about them.<br />
<br />
Similar to above, a full system may have an automated install process deploying on many machines. Or an initial virtual machine image is being used multiple times. (Do any hosting companies make any guarantees about the uniqueness of a VM you purchase from them? Are preinstalled private keys identical for all customers?)<br />
<br />
A system may be designed to use a seed file, a file which contains some random data, which is stored on disk to ensure a random state for next boot-up. The system may be rebooted at some point after the seed file is used, but before it is updated, causing an identical boot state.<br />
<br />
There can be multiple users on a system, which can influence or retrieve random data. Other users may very well know the sequences generated before and after some data was generated for another user. That can then be used in turn to determine what random data was generated for the other user.<br />
<br />
Hardware to generate random data may contain a backdoor, and should not be fully trusted.<br />
<br />
Hardware to generate random data may break, or be influenced in some manner, causing the generated data to not actually be random.<br />
<br />
Two machines next to each other may be measuring the same environmental conditions, leading one machine to know the random data being generated for the other.<br />
<br />
A process may be inheriting its state from its parent, where both of them will be generating the same sequences, or multiple children will be generating the same sequences.<br />
<br />
Wall-clock time can be adjusted, allowing the same time to occur multiple times, which in turn will lead to identical random states where wall-clock time is the only differentiating factor. <br />
<h4 style="text-align: center;">
Possible Solutions</h4>
A proper solution for identical manufacturing or deployments would be to write a unique initial state to each one. However, this cannot be relied upon, as it requires compliance by the manufacturer or deployer, which can increase costs and complexity, and good tools to do so are not available. <br />
<br />
Devices generally have components which contain serial numbers. These serial numbers should be mixed into initial states, minimally ensuring that identical devices do not have identical states. As an example, routers will have different <a href="http://en.wikipedia.org/wiki/MAC_address">MAC addresses</a>. They may even have multiple MAC addresses for different sides of the router, or for wireless. Be aware however that it is possible for an outsider to collect all the MAC addresses, and thus reveal the initial state.<br />
<br />
If a hardware clock is available, it should be mixed into the boot-up state to differentiate the boot-up state from previous boots and other identical machines that were booted at different times.<br />
<br />
Random devices should not emit data until some threshold of assurances are reached. Linux and NetBSD provide an API for checking certain thresholds, although race conditions make the API a bit useless for anything other than ensuring the devices are past an initial boot state. FreeBSD now ensures its random device does not emit anything till some assurances are met, but older versions did not carry this guarantee. Also be wary of relying on <i>/dev/random</i> for assurances here, as my prior random article demonstrated that the device under that path may be swapped for <i>/dev/urandom</i>, a practice done by many sysadmins or device manufacturers.<br />
<br />
Seed-files should not solely be relied upon, as embedded devices may not have them, in addition to the reboot issue. Above techniques should help with this.<br />
<br />
The initial random state provided to applications should be different for each user and application, so they are not all sharing data in the same sequence.<br />
<br />
Hardware generators should not be used without some pattern matching and statistical analysis to ensure they are functioning properly. Further, what they generate should only be used as a minor component towards random data generation, so they cannot solely influence the output, and prevent machines in the same environment producing the same results.<br />
<br />
Due to application states being cloned by<i> <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html">fork()</a></i> (or similar), a system wide random interface can be more secure than an application state (thus <a href="http://insanecoding.blogspot.com/2014/04/common-libressl-porting-mistakes.html">OpenSSL when directly using <i>/dev/urandom</i> can be more secure than various flavors of LibreSSL</a>). For application level random interfaces, they should have their states reset upon <i>fork()</i>. Mixing in time, process ID, and parent process ID can help.<br />
<br />
Always use the most accurate time possible. Also, mix in the amount of time running (which cannot be adjusted), not just wall-clock time.<br />
<br />
Many of the above solutions alone do not help much, as such data is readily known by others. This is a common library mistake where libraries tend to try system random devices, and if that fails, fall back on some of these other sources of uniquish data. Rather, system random devices should be used, which mix in all these other sources of uniquish data, and upon failure, go into <i>failure mode</i>, not <i>we can correct this extremely poorly mode</i>.<br />
<h4 style="text-align: center;">
Usage scenarios</h4>
Not only does random data have to be generated correctly, it has to be used correctly too.<br />
<br />
A common scenario is attempting to get a specific amount of random values. The typical approach is to divide the random value returned by the amount of possible values needed, and take the remainder. Let me demonstrate why this is wrong. Say your random number generator can generate 32 possible values, meaning any number from 0 to and including 31, and we only need to choose between five possible values.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-tlwMhFJv5y8/U3uL7qgldbI/AAAAAAAAAMA/eI4cqxVsAhA/s1600/modulo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-tlwMhFJv5y8/U3uL7qgldbI/AAAAAAAAAMA/eI4cqxVsAhA/s1600/modulo.png" /></a></div>
<br />
As can be seen from this chart, modulo reducing the random number will result in 0 and 1 being more likely than 2, 3, 4. Now if we were to suppose lower numbers are inherently better for some reason, we would consider such usage fine. But in that scenario, our random number generator should be:<br />
<blockquote class="tr_bq">
function GetInherentlyGoodRandomNumber()<br />
{<br />
return 0 //Lower numbers are inherently better!<br />
}</blockquote>
Being that the above is obviously ridiculous, we want to ensure equal distribution. So in the above scenario, if 30 or 31 occurs, a new random number should be chosen.<br />
<br />
Similar issues exist with <a href="http://en.wikipedia.org/wiki/Floating_point">floating point</a> random numbers. One generally uses floating point random numbers when one wants to use random values with certain probabilities. However, typical algorithms do not provide good distributions. There is an <a href="http://allendowney.com/research/rand/downey07randfloat.pdf">entire paper on this topic</a>, which includes some solutions.<br />
<br />
The C library's<i> <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html">fork()</a></i> call (when it exists) should be resetting any random state present in C library random functions, but unfortunately that's not done in the implementations I've checked (hello arc4random). The issue extends to external libraries, which most of them never even considered supplying <i>secure_fork()</i> or similar. Be wary to never use <i>fork()</i> directly. You almost always want a wrapper which handles various issues, not just random states.<br />
<h4 style="font-weight: normal; text-align: center;">
<b>Further reading</b></h4>
Pretty much everything I stated here is a summary or rehash of information presented in important books and papers. Now based on my past experiences that I described above, you're probably not going to read anything further. However, if the topics here apply to you, you really should read them.<br />
<ul>
<li><a href="http://www.amazon.com/gp/product/0596003943/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0596003943&linkCode=as2&tag=insacodi-20&linkId=JDS6WYNVV53NFMKD">Secure Programming Cookbook</a></li>
<li><a href="http://www.acsac.org/2003/papers/79.pdf">Practical Random Number Generation</a></li>
<li><a href="http://allendowney.com/research/rand/downey07randfloat.pdf">Generating Pseudo-random Floating-Point Values</a></li>
<li> <a href="http://www.amazon.com/gp/product/0470474246/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470474246&linkCode=as2&tag=insacodi-20">Cryptography Engineering</a></li>
<li><a href="https://www.factorable.net/weakkeys12.conference.pdf">Weak Keys</a></li>
<li><a href="https://factorable.net/weakkeys12.extended.pdf">Weak Keys Extended</a> </li>
</ul>
<br />
<br />insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com34tag:blogger.com,1999:blog-833174317742362874.post-7346711296403330732014-05-17T17:12:00.001-07:002014-05-17T17:14:45.414-07:00Protecting private keysWeb servers use <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">private keys</a> which they alone have in order to secure connections with users. Private keys must be protected at all costs.<br />
<br />
In order to protect private keys on disk, one generally encrypts them with a password, which is then needed by the web server upon launch in order to decrypt and use it in memory. However, if measures aren't taken to secure the memory containing the private key, it can be stolen from there too, which would be catastrophic.<br />
<br />
Normally, one doesn't need to worry about outsiders getting a hold of data from memory unless the attackers have direct access to the server itself. But bugs like <a href="http://en.wikipedia.org/wiki/Heartbleed">Heartbleed</a> allow remote users to grab random data from memory. Once data is in memory, the application and all the libraries it uses could divulge secret data if there is a buffer overflow lurking somewhere.<br />
<br />
To protect against exploiting such bugs, one should ensure that buffer overflows do not have access to memory containing private data. The memory containing private keys and similar kinds of data should be protected, meaning nothing should be allowed to read from them, not even the web server itself.<br />
<br />
Now obviously a program needs some access to a private key in order to work with it, so it can't just prevent all access from it. Rather, once a private key or similar data is loaded into memory, that memory should have its read permissions removed. When, and only when some activity needs to be performed with the private key, read permissions can be restored, the activity performed, and then read permissions revoked. This will ensure the rest of the application cannot access what it does not need, nor should be allowed to access.<br />
<br />
On UNIX systems, one can use <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/mprotect.html">mprotect()</a></i> to change the permission on a page of memory. On Windows, one can use <i><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa366898%28v=vs.85%29.aspx">VirtualProtect()</a></i>.<br />
<br />
The above however has a crucial flaw - <a href="http://en.wikipedia.org/wiki/Thread_%28computing%29">multi-threading</a>. In a threaded application, all threads have access to data of other threads. So while one thread may be performing some critical private key related code, and allows read access for the moment, another thread can read it outside of the critical portion of code too. Therefore, even more isolation is needed.<br />
<br />
To truly isolate the code that uses a private key and similar data, all the code that handles that stuff should be placed into its own <a href="http://en.wikipedia.org/wiki/Process_%28computing%29">process</a>. The rest of the application can then request that well defined activities be performed via a <a href="http://en.wikipedia.org/wiki/Anonymous_pipe">pipe</a> or other form of <a href="http://en.wikipedia.org/wiki/Inter-process_communication">inter-process communication</a>. This will also ensure that other kinds of bugs in the application, such as buffer overflows that allow <a href="http://en.wikipedia.org/wiki/Arbitrary_code_execution">arbitrary code execution</a> cannot reestablish read access to the secret data.<br />
<br />
On UNIX systems, one can use <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html">fork()</a> to create a process which is still part of the same application. On all systems, the separate process can be a separate application with a well defined restrictive IPC API with limited and secured access by the web server.<br />
<br />
No insecure library or silly bug in your web server should ever allow the application to divulge such secrets. If such services are not utilizing the techniques above, then they're just biding their time until the next Heartbleed.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com7tag:blogger.com,1999:blog-833174317742362874.post-88755393666525325002014-05-10T15:04:00.000-07:002014-05-10T15:10:16.874-07:00Copying code != Copying implementationA little over a week ago, I wrote an article about some <a href="http://insanecoding.blogspot.com/2014/04/common-libressl-porting-mistakes.html">common porting mistakes</a> for a new library - LibreSSL. I've since received multiple questions about copying code from other projects. Mainly, if the original project uses certain code or another popular library is using some compatibility code, what is wrong with copying one of those directly?<br />
<br />
The most obvious problem with copying from some compatibility library is that <a href="http://insanecoding.blogspot.com/2007/03/are-open-source-libraries-written.html">they may not be written properly</a>. <b>The primary concern behind most compatibility libraries is that things should compile</b>. Just because things compile doesn't mean errors and all scenarios are being handled properly, or that the implementation is in any way secure. The popular <i>libbsd</i> and many GNU library shims may seem to get the job done, but only superficially.<br />
<br />
However, the real problem is that <b>copying some code does not mean you copied the implementation</b>. There's a lot more to an implementation than just some code. For example, perhaps the <b><a href="http://insanecoding.blogspot.com/2014/04/libressl-good-and-bad.html">source in question is being compiled with certain parameters or with special compilers</a></b>, and the code is written specifically for that situation. If you copied the code but not the build environment, you only nabbed part of the implementation.<br />
<br />
Another point to consider is if the code you copied is even used how you think it is. Let's look at a function from <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/common/lib/libc/string/consttime_memequal.c?rev=1.4&content-type=text/x-cvsweb-markup">NetBSD</a>:<br />
<br />
<blockquote class="tr_bq">
<pre>int
consttime_memequal(const void *b1, const void *b2, size_t len)
{
const char *c1 = b1, *c2 = b2;
int res = 0;
while (len --)
res |= *c1++ ^ *c2++;
<b> /*
* If the compiler for your favourite architecture generates a
* conditional branch for `!res', it will be a data-dependent
* branch, in which case this should be replaced by
*
* return (1 - (1 & ((res - 1) >> 8)));
*
* or <u>rewritten in <i>assembly</i></u>.
*/</b>
return !res;
}</pre>
</blockquote>
Note how the comment here says that this code may not function correctly on all architectures. In fact, <u>some architectures may have an implementation in assembly which works correctly, whereas the C version does not</u>.<br />
<br />
NetBSD, like most C libraries, offers assembly variants for certain functions on certain architectures. If you copy a C file from it without the assembly files for the architecture in question, then the implementation was not copied, and in fact, may even produce incorrect results. You can't always depend on scanning comments either, as <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/common/lib/libc/string/consttime_memequal.c?rev=1.3&content-type=text/x-cvsweb-markup">previous versions of this C file didn't contain the comment</a>.<br />
<br />
Now, let's say you copied files and their <b>build environment</b>, including possible assembly variants, does that now mean the implementation was definitively copied? No! Some functions used in the copied code may actually depend on certain characteristics within this particular implementation. Let me provide an example. Let's say <i>consttime_memequal()</i> was implemented as follows:<br />
<br />
<blockquote class="tr_bq">
int consttime_memequal(const void *s1, const void *s2, size_t n)<br />
{<br />
return !memcmp(s1, s2, n);<br />
} </blockquote>
<i>consttime_memequal()</i> must run in a constant amount of time regardless if whether its two parameters are equal or not. Yet this version here wraps directly to <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/memcmp.html">memcmp()</a></i>, which does not make such guarantees. Assuming this version was correct on the platform in question, <b>it is correct because of an external factor</b>, not because of the C code presented here.<br />
<br />
Some compilers may provide their own version of <i>memcmp()</i> which they'll use instead of the C library's, such as <a href="http://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html">GCC</a>. Perhaps some compiler provides <i>memcmp()</i>, along with a compile option to make all <i>memcmp()</i> operations constant-time. The file containing this function can then be compiled with constant-time-memcmp enabled, and now everything else can call <i>consttime_memequal()</i> without any special compilers or build options.<br />
<br />
Now while the above is possible, it's not the only possibility. Perhaps <i>memcmp()</i> on the platform in question just happens to be constant-time for some reason. In such a scenario, <b>even if you used the exact same compiler with the exact same build options, and the exact same code, your implementation is still incorrect</b>, because your underlying functions behave differently.<br />
<br />
Bottom line, <b>just copying code is incorrect</b>. If some function is supposed to be doing something with certain properties beyond just fulfilling some activity (perhaps constant time, or defying compiler optimizations), then <b>you must review the entire implementation and determine what aspect of it gives it this property</b>. Without doing so, you're only fooling yourself into believing you copied an implementation.<br />
<br />
Implementation copying checklist:<br />
<ul>
<li>Understand what you're attempting to copy, and what properties it carries. </li>
<li>Ensure what you're copying actually performs its objectives.</li>
<li>Ensure you copy the entire implementation.</li>
<li>Ensure there's nothing about the compiler, compile options, or other aspects of the build environment which you forgot to copy.</li>
<li>Ensure you copy any needed alternatives for certain architectures.</li>
<li>Ensure the code does not depend on different implementations of functions you already have.</li>
<li>Test everything for correctness, including matching output for error situations, and extra properties.</li>
</ul>
insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com14tag:blogger.com,1999:blog-833174317742362874.post-25039600643680949752014-05-03T22:04:00.000-07:002014-05-03T22:50:57.794-07:00A good idea with bad usage: /dev/urandomLast week, I wrote <a href="http://insanecoding.blogspot.com/2014/04/libressl-good-and-bad.html">two</a> <a href="http://insanecoding.blogspot.com/2014/04/common-libressl-porting-mistakes.html">articles</a> pointing out issues with unofficial porting efforts of <a href="http://www.libressl.org/">LibreSSL</a>. In these articles, I highlighted some issues that I currently see going on with some of these projects.<br />
<br />
In the second article, I called attention to poor <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=arc4random_buf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">arc4random_buf()</a> implementations being created, specifically saying: <i>"</i><span dir="auto"><i>Using poor sources of <a href="http://en.wikipedia.org/wiki/Entropy_%28computing%29">entropy</a> like </i><i>/dev/urandom on Linux, or worse, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/gettimeofday.html">gettimeofday()</a>, and using them to generate long-lived keys."</i> Which seemed to irk some. Now for those that care to understand the quoted issue in its proper context should go look for the current LibreSSL porting projects they can find via Google, and review their source. However, most seem to be understanding this as some kind of argument about </span><span dir="auto"><i>/dev/random</i> </span>versus <span dir="auto"><i>/dev/urandom</i></span>. It isn't. So without further ado:<br />
<h3 style="text-align: center;">
Poorly seeding a <span dir="auto"><a href="http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">cryptographically secure pseudorandom number generator</a> (CSPRNG)</span></h3>
<span dir="auto">In order for many forms of cryptography to work properly, they depend on secrecy, unpredictability, and uniqueness. Therefore, we need a good way to use many unpredictable values.</span><br />
<br />
<span dir="auto">Now randomness doesn't really exist. When we humans see something as random, it's only because we don't know or understand all the details. Therefore, </span><span dir="auto"><i>any perceived randomness on your part is your inability to track all the variables</i>.</span><br />
<br />
<span dir="auto">Computers are very complex machines, where many different components are all working independently, and in ways that are hard to keep track of externally. Therefore, the operating system is able to collect variables from all the various hardware involved, their current operating conditions, how they handle certain tasks, how long they take, how much electricity they use, and so on. These variables can now be combined together using very confusing and ridiculous algorithms, which essentially throw all the data through a washing machine and the world's worst roller coaster. This result is what we call <i>entropy</i>.</span><br />
<span dir="auto"><br /></span>
<span dir="auto">Now entropy might not be that large, and can only provide a small amount of unique <i>random</i> values to play with. However, a small amount of random values can be enough to create trillions of <a href="http://en.wikipedia.org/wiki/Pseudorandomness">pseudo-random</a> values. To do so, one uses some of the aforementioned ridiculous algorithms to produce two values. One value is never seen outside the pseudo-random number generator, and is used as part of the calculations for the next time a random value is requested, and the other value is output as the pseudo-random value generated for a single request.</span><br />
<span dir="auto"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-TAnxx8Zmuv4/U2WMgdbpYyI/AAAAAAAAALw/Ywe30fLxidA/s1600/csprng.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-TAnxx8Zmuv4/U2WMgdbpYyI/AAAAAAAAALw/Ywe30fLxidA/s1600/csprng.png" height="274" width="320" /></a></div>
<span dir="auto"><br /></span>
<span dir="auto"><br /></span>
<span dir="auto">A construct of this nature can allow an unlimited amount of <i>"random"</i> values to be generated. As long as this technique never repeats and is unpredictable, then it is cryptographically secure. Of course since the algorithm is known, the entropy <i>seeding</i> it must be a secret, otherwise it is completely predictable.</span><br />
<span dir="auto"><br /></span>
<span dir="auto">Different algorithms have different properties. OpenBSD's arc4random set of functions are known to be able to create a very large amount of good random values from a very little amount of entropy. Of course the better entropy it is supplied with, the better it can perform, so you'll always want the best entropy possible. As with any random number generator, supply it with predictable values, and its entire security is negated. </span><br />
<h4 style="text-align: center;">
<span dir="auto">arc4random and </span><span dir="auto">/dev/(u)random </span></h4>
<span dir="auto">So, how does one port the arc4random family to Linux? Linux is well known for inventing and supplying two <i>default</i> files, <i>/dev/random</i> and <i>/dev/urandom</i> (unlimited random). The former is pretty much raw entropy, while the latter is the output of a CSPRNG function like OpenBSD's arc4random family. The former can be seen as more random, and the latter as less random, but the differences are extremely hard to measure, which is why CSPRNGs work in the first place. Since the former is only entropy, it is limited as to how much it can output, and one needing a lot of random data can be stuck waiting a while for it to fill up the random buffer. Since the latter is a CSPRNG, it can keep outputting data indefinitely, without any significant waiting periods.</span><br />
<br />
<span dir="auto">Now theoretically, one can make the <i>arc4random_buf()</i> function a wrapper around <i>/dev/urandom</i>, and be done with it. The only reason not to is because one may trust the arc4random set of algorithms more than <i>/dev/urandom</i>. In that case, would<i> /dev/urandom</i> be trusted enough to seed the arc4random algorithms, which is then in turn used for many other outputs, some of which end up in RSA keys, SSH keys, and so on? I'll leave that question to the cryptographic experts. But I'll show you how to use <i>/dev/urandom</i> poorly, how to attack the design, and corrective efforts.</span><br />
<span dir="auto"><br /></span>
<span dir="auto">First, take a look at how <a href="https://github.com/busterb/libressl/blob/089aaf7ad3c4c4c32d216b05e69a83d0f2f75140/crypto/compat/arc4random.c#L96">one project</a> decided to handle the situation. It tries to use <i>/dev/urandom</i>, and in a worse case scenario uses <i>gettimeofday()</i> and other predictable data. Also, in a case where the </span><i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html">read()</a></i><span dir="auto"> function doesn't return as much as requested for some reason, but perhaps returned a decent chunk of it, the <i>gettimeofday()</i> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/getpid.html">getpid()</a> calls will overwrite what was returned.</span><span dir="auto"><i></i></span><br />
<br />
<span dir="auto">This very much reminds me of why the OpenBSD team <a href="http://freshbsd.org/commit/openbsd/58777eed1cff7c5b34cbc026278f730176a6dbc2?diff=lib%2Flibssl%2Fsrc%2Fcrypto%2Frand%2Frand_unix.c">removed the OpenSSL techniques</a> in the first place. You do not want to use a time function as your primary source of entropy, nor with range limited values, and other dumb things. If you're going to use time, at the very least </span><span dir="auto">use <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_getres.html">clock_gettime()</a></i> which provides time resolution that is 1,000 times better than </span><span dir="auto"><i>gettimeofday()</i></span><span dir="auto">, and can provide both current time, and monotonic time (time which cannot go backwards). Additionally, both <i>gettimeofday()</i> and <i>clock_gettime()</i> on 64-bit systems will return values where the top half of their data will be zeroed out, so you'll want to ensure you throw that away, and not just use their raw data verbatim. Further, how this one project uses <i>/dev/urandom</i>, like most other projects, is terrible.</span><br />
<h4 style="text-align: center;">
<span dir="auto">Attacking /dev/(u)random usage</span></h4>
<span dir="auto">A naive approach to use /dev/urandom (and /dev/random) is as follows:</span><br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> fd = open(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">, O_RDONLY);</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fd != -</span><span style="color: blue;">1</span><span style="color: black;">)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (read(fd, buffer, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)) == </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)))</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Do what needs to be done with the random data...</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Read error handling</span>
<span style="color: black;"> }</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;">}</span>
<span style="color: black; font-weight: bold;">else</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Open error handling</span>
<span style="color: black;">}</span></pre>
<br />
This tries to open the file, ensures it was opened, tries to read 40 bytes of data, and continues what it needs to do in each scenario. Note, 40 bytes is the amount arc4random wants, which also happens to be 8 bytes more than the Linux manual page says you should be using at a time from its random device.<br />
<br />
The first common mistake here is using <i>read()</i> like this. <i>read()</i> can be interrupted. Normally it can't be interrupted for regular files, but this device is not a regular file. Some random device implementations specifically document that <i>read()</i> can be interrupted upon them.<br />
<br />
So now our second attempt: <br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: grey; font-style: italic;">//Like read(), but keep reading upon interuption, until everything possible is read</span>
<span style="color: black;">ssize_t insane_read(</span><span style="color: maroon;">int</span><span style="color: black;"> fd, </span><span style="color: maroon;">void</span><span style="color: black;"> *buf, size_t count)</span>
<span style="color: black;">{</span>
<span style="color: black;"> ssize_t amount_read = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">while</span><span style="color: black;"> ((size_t)amount_read < count)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> ssize_t r = read(fd, (</span><span style="color: maroon;">char</span><span style="color: black;"> *)buf+amount_read, count-amount_read);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (r > </span><span style="color: blue;">0</span><span style="color: black;">) { amount_read += r; }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!r) { </span><span style="color: black; font-weight: bold;">break</span><span style="color: black;">; }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (errno != EINTR)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> amount_read = -</span><span style="color: blue;">1</span><span style="color: black;">;</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">break</span><span style="color: black;">;</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(amount_read);</span>
<span style="color: black;">}</span>
<span style="color: maroon;">int</span><span style="color: black;"> success = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: maroon;">int</span><span style="color: black;"> fd = open(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">, O_RDONLY);</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fd != -</span><span style="color: blue;">1</span><span style="color: black;">)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;"> ssize_t amount = insane_read(fd, buffer, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)); </span><span style="color: grey; font-style: italic;">//Grab as much data as we possibly can</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount > </span><span style="color: blue;">0</span><span style="color: black;">)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount < </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer))</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Continue filling with other sources</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Do what needs to be done with random data...</span>
<span style="color: black;"> success = </span><span style="color: blue;">1</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Yay!</span>
<span style="color: black;"> }</span>
<span style="color: black;">}</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!success)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Error handling</span>
<span style="color: black;">}</span></pre>
<br />
With this improved approach, we know we're reading as much as possible, and if not, then we can try using lesser techniques to fill in the missing entropy required. So far so good, right?<br />
<br />
Now, let me ask you, why would opening <i>/dev/(u)random</i> fail in the first place? First, it's possible the open was interrupted, as may happen on some implementations. So the <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html">open()</a></i> call should probably be wrapped like <i>read()</i> is. In fact, you may consider switching to the C family <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html">fopen()</a></i> and <i><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fread.html">fread()</a></i> calls which handle these kinds of problems for you. However, opening could be failing because the application has reached its <a href="http://en.wikipedia.org/wiki/File_descriptor">file descriptor</a> limit, which is even more prevalent with the C family of file functions. Another possibility is that the file doesn't even exist. Go ahead, try to delete the file as the <a href="http://en.wikipedia.org/wiki/Superuser">superuser</a>, nothing stops you. Also, you have to consider that applications may be running inside a <a href="http://en.wikipedia.org/wiki/Chroot">chroot</a>, and <i>/dev/</i> entries don't exist.<br />
<br />
I'll cover some alternative approaches for the above problems later. But if you managed to open and read all the data needed, everything is great, right? Wrong! How do you even know if <i>/dev/(u)random</i> is random in the first place? This may sound like a strange question, but it isn't. You can't just trust a file because of it's path. Consider an attacker ran the following:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">void</span><span style="color: black;"> sparse_1gb_overwrite(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *path)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> fd;</span>
<span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> byte = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//No error checking</span>
<span style="color: black;"> unlink(path);</span>
<span style="color: black;"> fd = open(path, O_CREAT | O_WRONLY | O_TRUNC, </span><span style="color: teal;">0644</span><span style="color: black;">);</span>
<span style="color: black;"> lseek(fd, </span><span style="color: blue;">1073741822</span><span style="color: black;">, SEEK_SET);</span>
<span style="color: black;"> write(fd, &byte, </span><span style="color: blue;">1</span><span style="color: black;">);</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;">}</span>
<span style="color: black;">sparse_1gb_create(</span><span style="color: #dd0000;">"/dev/random"</span><span style="color: black;">);</span>
<span style="color: black;">sparse_1gb_create(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">);</span>
</pre>
<br />
Now both random <i>devices</i> are actually large <a href="http://en.wikipedia.org/wiki/Sparse_file">sparse files</a> with known data. This is worse than not having access to these files, in fact, the attacker is able to provide you with a seed of his own choosing! A strong cryptographic library should not just assume everything is in a proper state. In fact, if you're using a <i>chroot</i>, you're already admitting you don't trust what will happen on the file system, and you want to isolate some applications from the rest of the system.<br />
<br />
So the next step is to ensure that the so called device you're opening is actually a device:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> success = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: maroon;">int</span><span style="color: black;"> fd = insane_open(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">, O_RDONLY);</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fd != -</span><span style="color: blue;">1</span><span style="color: black;">)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">struct</span><span style="color: black;"> stat stat_buffer;</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!fstat(fd, &stat_buffer) && S_ISCHR(stat_buffer.st_mode)) </span><span style="color: grey; font-style: italic;">//Make sure we opened a character device!</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;"> ssize_t amount = insane_read(fd, buffer, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)); </span><span style="color: grey; font-style: italic;">//Grab as much data as we possibly can</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount > </span><span style="color: blue;">0</span><span style="color: black;">)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount < </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer))</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Continue filling with other sources</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Do what needs to be done with random data...</span>
<span style="color: black;"> success = </span><span style="color: blue;">1</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Yay!</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;">}</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!success)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Error handling</span>
<span style="color: black;">}</span></pre>
<br />
<br />
So now we're out of the woods, right? Unfortunately not yet. How do you know you opened the correct character device? Maybe <i>/dev/urandom</i> is <a href="http://en.wikipedia.org/wiki/Symbolic_link">symlinked</a> to <a href="http://en.wikipedia.org/wiki//dev/zero">/<i>dev/zero</i></a>? You can run <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/lstat.html">lstat()</a> on <i>/dev/urandom</i> initially, but that has <a href="http://en.wikipedia.org/wiki/Time_of_check_to_time_of_use">TOCTOU</a> issues. We can add the FreeBSD/Linux extension <i>O_NOFOLLOW</i> to the open command, but then<i> /dev/urandom</i> can't be used when it's linked to <i>/dev/random</i> as it is on FreeBSD, or linked to some other location entirely as on Solaris. Furthermore, avoiding symlinks is not enough:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">void</span><span style="color: black;"> dev_zero_overwrite(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *path)</span>
<span style="color: black;">{</span>
<span style="color: black;"> unlink(path);</span>
<span style="color: black;"> mknod(path, S_IFCHR | </span><span style="color: teal;">0644</span><span style="color: black;">, makedev(</span><span style="color: blue;">1</span><span style="color: black;">, </span><span style="color: blue;">5</span><span style="color: black;">));</span>
<span style="color: black;">}</span>
<span style="color: black;">dev_zero_overwrite(</span><span style="color: #dd0000;">"/dev/random"</span><span style="color: black;">);</span>
<span style="color: black;">dev_zero_overwrite(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">);</span>
</pre>
<br />
If an attacker manages to run the above code on Linux, the random devices are now both in their very essence <i>/dev/zero</i>!<br />
<br />
Here's a list of device numbers for the random devices on various Operating Systems:<br />
<br />
<style type="text/css">
<!--
@page { margin: 0.79in }
TD P { margin-bottom: 0in }
P { margin-bottom: 0.08in }
</style>
<br />
<table cellpadding="4" cellspacing="0" style="width: 513px;">
<colgroup><col width="89"></col>
<col width="39"></col>
<col width="60"></col>
<col width="101"></col>
<col width="55"></col>
<col width="67"></col>
<col width="44"></col>
</colgroup><tbody>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="89"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="39">Linux</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="60">FreeBSD</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="101">DragonFlyBSD</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="55">NetBSD</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="67">OpenBSD</td>
<td style="border: 1px solid #000000; padding: 0.04in;" width="44">Solaris</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="89">/dev/random</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="39">1:8</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="60">0:10</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="101">8:3</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="55">46:0</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="67">45:0</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="44">0:0</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="89">/dev/urandom</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="39">1:9</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="60"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="101">8:4</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="55">46:1</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="67">45:2</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="44">0:1</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="89">/dev/srandom</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="39"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="60"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="101"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="55"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="67">45:1</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="44"><br />
<br /></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="89">/dev/arandom</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="39"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="60"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="101"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="55"><br />
<br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="67">45:3</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="44"><br />
<br /></td>
</tr>
</tbody></table>
<br />
If your application is running with Superuser privileges, you can actually create these random devices anywhere on the fly:<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> result = mknod(</span><span style="color: #dd0000;">"/tmp/myurandom"</span><span style="color: black;">, S_IFCHR | </span><span style="color: teal;">0400</span><span style="color: black;">, makedev(</span><span style="color: blue;">1</span><span style="color: black;">, </span><span style="color: blue;">9</span><span style="color: black;">)); </span><span style="color: grey; font-style: italic;">//Create "/dev/urandom" on Linux</span></pre>
<br />
Of course, after opening, you want to ensure that you're using what you think you're using:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> success = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: maroon;">int</span><span style="color: black;"> fd = insane_open(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">, O_RDONLY);</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fd != -</span><span style="color: blue;">1</span><span style="color: black;">)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">struct</span><span style="color: black;"> stat stat_buffer;</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!fstat(fd, &stat_buffer) && S_ISCHR(stat_buffer.st_mode) &&</span>
<span style="color: black;"> ((stat_buffer.st_rdev == makedev(</span><span style="color: blue;">1</span><span style="color: black;">, </span><span style="color: blue;">8</span><span style="color: black;">)) || (stat_buffer.st_rdev == makedev(</span><span style="color: blue;">1</span><span style="color: black;">, </span><span style="color: blue;">9</span><span style="color: black;">)))) </span><span style="color: grey; font-style: italic;">//Make sure we opened a random device</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;"> ssize_t amount = insane_read(fd, buffer, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)); </span><span style="color: grey; font-style: italic;">//Grab as much data as we possibly can</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount > </span><span style="color: blue;">0</span><span style="color: black;">)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount < </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer))</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Continue filling with other sources</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Do what needs to be done with random data...</span>
<span style="color: black;"> success = </span><span style="color: blue;">1</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Yay!</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;">}</span>
</pre>
<br />
The Linux user manual page for the random devices explicitly informs the reader of these magic numbers, so hopefully they won't change. I have no official sources for the magic numbers on the other OSs. Now, you'll notice that I checked here that the file descriptor was open to either Linux random device. A system administrator may for some reason replace one with the other, so don't necessarily rely on a proper system using the expected device under the device name you're trying to use.<br />
<br />
This brings us back to <i>/dev/random</i> versus <i>/dev/urandom</i>. Different OSs may implement these differently. On FreeBSD for example, there is only the former, and it is a CSPRNG. MirBSD offers five different random devices with all kinds of semantics, and who knows how a sysadmin may shuffle them around. On Linux and possibly others, <i>/dev/urandom</i> has a fatal flaw that in fact it may not be seeded properly, so blind usage of it isn't a good idea either. Thankfully Linux and NetBSD offer the following:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> data;</span>
<span style="color: maroon;">int</span><span style="color: black;"> result = ioctl(fd, RNDGETENTCNT, &data); </span><span style="color: grey; font-style: italic;">//Upon success data now contains amount of entropy available in bits</span>
</pre>
<br />
This <i>ioctl()</i> call will only work on a random device, so you can use this instead of the <i>fstat()</i> call on these OSs. You can then check data to ensure there's enough entropy to do what you need to:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: maroon;">int</span><span style="color: black;"> success = </span><span style="color: blue;">0</span><span style="color: black;">;</span>
<span style="color: maroon;">int</span><span style="color: black;"> fd = insane_open(</span><span style="color: #dd0000;">"/dev/urandom"</span><span style="color: black;">, O_RDONLY);</span>
<span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fd != -</span><span style="color: blue;">1</span><span style="color: black;">)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> entropy;</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!ioctl(fd, RNDGETENTCNT, &entropy) && (entropy >= (</span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer) * </span><span style="color: blue;">8</span><span style="color: black;">))) </span><span style="color: grey; font-style: italic;">//This ensures it's a random device, and there's enough entropy</span>
<span style="color: black;"> {</span>
<span style="color: black;"> ssize_t amount = insane_read(fd, buffer, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer)); </span><span style="color: grey; font-style: italic;">//Grab as much data as we possibly can</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount > </span><span style="color: blue;">0</span><span style="color: black;">)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (amount < </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer))</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Continue filling with other sources</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Do what needs to be done with random data...</span>
<span style="color: black;"> success = </span><span style="color: blue;">1</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Yay!</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> close(fd);</span>
<span style="color: black;">}</span>
</pre>
<br />
However, there may be a TOCTOU between the <i>ioctl()</i> and the <i>read()</i> call, I couldn't find any data on this, so take the above with a grain of salt. This also used to work on OpenBSD too, but they removed the ioctl() command <i>RNDGETENTCNT</i> a couple of versions back.<br />
<br />
Linux has one other gem which may make you want to run away screaming. Look at the following from the manual for random device <i>ioctl()</i>s:<br />
<br />
<blockquote class="tr_bq">
RNDZAPENTCNT, RNDCLEARPOOL<br />Zero the entropy count of all pools and add some system data (such as wall clock) to the pools.</blockquote>
If that last one does what I think it does, is any usage ever remotely safe?<br />
<h4 style="text-align: center;">
/dev/(u)random conclusion </h4>
<br />
After all this, we now know the following:<br />
<ul>
<li>This is hardly an interface which is easy to use correctly (and securely).</li>
<li>On some OSs there may not be any way to use it correctly (and securely). </li>
<li>Applications not running with Superuser privileges may have no way to access the random device.</li>
<li>An application at its file descriptor limit cannot use it.</li>
<li>Many portability concerns. </li>
</ul>
<br />
For these reasons, OpenBSD created <i>arc4random_buf()</i> in the first place. It doesn't suffer from these above problems. The other BSDs also copied it, although they may be running on older less secure implementations.<br />
<h4 style="text-align: center;">
Alternatives</h4>
<br />
In addition to a CSPRNG in <a href="http://en.wikipedia.org/wiki/User_space">userspace</a>, the BSDs also allow for a way to get entropy directly from the kernel:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#define NUM_ELEMENTS(x) (sizeof(x)/sizeof((x)[0]))</span>
<span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;">size_t len = </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer);</span>
<span style="color: maroon;">int</span><span style="color: black;"> mib[] = { CTL_KERN, KERN_RND };</span>
<span style="color: maroon;">int</span><span style="color: black;"> result = sysctl(mib, NUM_ELEMENTS(mib), buffer, &len, </span><span style="color: blue;">0</span><span style="color: black;">, </span><span style="color: blue;">0</span><span style="color: black;">);</span></pre>
<br />
<br />
<i>KERN_RND</i> may also be replaced with<i> KERN_URND</i>, <i>KERN_ARND</i>, and others on the various BSDs. FreeBSD also has a similar interface to determine if the kernel's CSPRNG is properly seeded, see the FreeBSD user manual page for more details. DragonFlyBSD also provides <i>read_random()</i> and <i>read_random_unlimited()</i> as direct no-nonsense interfaces to the underlying devices for <i>/dev/random</i> and <i>/dev/urandom</i>.<br />
<br />
Now Linux used to provide a similar interface as the BSDs:<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#define NUM_ELEMENTS(x) (sizeof(x)/sizeof((x)[0]))</span>
<span style="color: maroon;">uint8_t</span><span style="color: black;"> buffer[</span><span style="color: blue;">40</span><span style="color: black;">];</span>
<span style="color: black;">size_t len = </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(buffer);</span>
<span style="color: maroon;">int</span><span style="color: black;"> mib[] = { CTL_KERN, KERN_RANDOM, RANDOM_UUID };</span>
<span style="color: maroon;">int</span><span style="color: black;"> result = sysctl(mib, NUM_ELEMENTS(mib), buffer, &len, NULL, </span><span style="color: blue;">0</span><span style="color: black;">);</span>
</pre>
<br />
However, this interface was removed a couple of versions back. Kind of makes you wonder why there is no sane simple to use standardized API everywhere that doesn't depend on a house of cards. This is the kind of thing you would think would be standardized in POSIX.<br />
<h4 style="text-align: center;">
Conclusion</h4>
<br />
There does not seem to be any good cross-platform techniques out there. Anything done with the current technology will probably require a ton of conditional compiles, annoying checks, and ridiculous workarounds. Although, there should be enough basis here to come up with something with a high confidence level. It'd be interesting to see what the LibreSSL team (or the OpenSSH team) comes up with when they get around to porting what's needed here. As I said before, until then, avoid the non-official ports out there, <span dir="auto">which use poor sources of entropy like an insecure use of <i>/dev/urandom</i> on Linux, which then falls back on <i>gettimeofday()</i>, and is used to generate long-lived keys.</span><br />
<br />insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com212tag:blogger.com,1999:blog-833174317742362874.post-67322129438069529402014-04-30T09:42:00.000-07:002014-05-02T05:12:21.004-07:00Common LibreSSL porting mistakesThe other day I wrote an <a href="http://insanecoding.blogspot.com/2014/04/libressl-good-and-bad.html">article</a> discussing <a href="http://www.libressl.org/">LibreSSL</a>, and the common mistakes being made by those thinking they know how to port it to other platforms. Since then, I've seen even more non-official ports pop up, and even some discussions about various Linux distros or other projects switching from OpenSSL to LibreSSL via one of the ports. I've also gotten multiple requests to elaborate a bit on some of the most common mistakes I'm seeing across these porting projects.<br />
<br />
So here is a more verbose explanation of some of the most common problems I'm seeing:<br />
<h4 style="text-align: center;">
<i><a href="http://www.openbsd.org/cgi-bin/man.cgi?query=explicit_bzero&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">explicit_bzero()</a></i> isn't.</h4>
This function needs to ensure it <u>cannot be optimized out</u>. However, several projects are either using macros to define <i>explicit_bzero</i> as <i>bzero</i>, or are wrapping <i>explicit_bzero()</i> to <i>bzero()</i> <u>without using any optimization parameters</u> which will ensure the function call stays in. Especially problematic in the case of using link-time optimizations (LTO).<br />
<h4 style="text-align: center;">
<i><a href="http://www.openbsd.org/cgi-bin/man.cgi?query=reallocarray&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">reallocarray()</a></i> directly wrapped to <i>realloc()</i></h4>
The former function receives the size of the element, and how many elements we're dealing with, and the latter just takes a raw amount. It's a good thing to use <i>reallocarray()</i> when you need to reallocate say a <i>struct whatever[5000]</i> to a <i>struct whatever[30000]</i>, so you can pass <i>sizeof(</i><i>struct whatever</i><i>)</i> and <i>30000</i> as two separate parameters, as opposed to calulcating <i>sizeof(</i><i>struct whatever</i><i>)*30000 </i><u>which may overflow</u>. But the naive implementations are just directly wrapping to <i>realloc(),</i> <b>reintroducing the problem</b> that <i>reallocarray()</i> is supposed to be fixing.<br />
<br />
There's also issues of <a href="http://en.wikipedia.org/wiki/Data_structure_alignment">alignment</a> to consider when blindly multiplying where small values are concerned, but I won't go into them here.<br />
<h4 style="text-align: center;">
Poor <i><a href="http://www.openbsd.org/cgi-bin/man.cgi?query=arc4random_buf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">arc4random_buf()</a></i> implementations</h4>
This function is supposed to fill a buffer using a <span dir="auto"><a href="http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">cryptographically secure pseudorandom number generator</a>. However, I'm seeing a whole class of dumbness here:</span><br />
<ul>
<li><span dir="auto">Using classical pseudorandom number generators.</span></li>
<li><span dir="auto">Using older less secure implementations.</span></li>
<li><span dir="auto">Using poor sources of <a href="http://en.wikipedia.org/wiki/Entropy_%28computing%29">entropy</a> like <i>/dev/urandom</i> on Linux, or worse, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/gettimeofday.html">gettimeofday()</a>, and using them to generate long-lived keys. </span> </li>
</ul>
<h4 style="text-align: center;">
OpenBSD functions may be more secure than counterparts elsewhere</h4>
<div style="text-align: left;">
This is a generic issue where OpenBSD is deleting some silly platform wrappers, or reducing multiple functions calls with glue logic down to a single standardized function. <b>OpenBSD is depending on the security of their implementation</b> of said function, while the porters have no idea that their platform is less secure, and have no inkling that something is wrong, because there are no compiler errors about missing functions in this scenario. One common case where this scenario is true is with the <i><a href="http://www.openbsd.org/cgi-bin/man.cgi?query=calloc&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">calloc()</a></i> function, where the OpenBSD implementation checks for issues, but <a href="http://kqueue.org/blog/2012/03/05/memory-allocator-security-revisited/">several other platforms unfortunately do not</a>. </div>
<h4 style="text-align: center;">
The future</h4>
I've actually predicted all the above problems would appear in ports once I saw what OpenBSD is doing to OpenSSL, before any of the ports of LibreSSL became public. There's a couple of other significant mistakes I'm expecting to see appear in LibreSSL ports, but have not seen yet. These probably already exists in ports I haven't reviewed, or will exist in the wild soon enough. Chief among them is implementing <i><a href="http://www.openbsd.org/cgi-bin/man.cgi?query=timingsafe_bcmp&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html">timingsafe_bcmp()</a></i>. I'm expecting to see implementations which directly wrap to regular <i>bcmp(),</i> which unlike the former, is not performed in constant-time, and can expose the application to <a href="http://en.wikipedia.org/wiki/Timing_attack">timing attacks</a>. I'm confident in this because it's not the first time I've seen OpenBSD projects ported, and all these issues exist till this day in various projects in the wild. If you're going to port a project, please take a moment to really understand the functions you're porting, reviewing the manpages and all the comments in the source code, and truly understanding the ramifications of what you're doing.<br />
<br />
As I said before, avoid any LibreSSL port which is not from the LibreSSL team itself, or from another team with a proven track record for knowing how to develop a secure environment.<br />
<br />
Lastly, here's some Google searches which may turn up some of the issues mentioned here, but is by no means exhaustive:<br />
<a href="https://www.google.com/search?q=%22-Dexplicit_bzero%3Dbzero%22">https://www.google.com/search?q="-Dexplicit_bzero%3Dbzero"</a><br />
<a href="https://www.google.com/search?q=%22%23define+explicit_bzero+bzero%22">https://www.google.com/search?q="%23define+explicit_bzero+bzero"</a><br />
<a href="https://www.google.com/search?q=%22-Dtimingsafe_bcmp%3Dbcmp%22">https://www.google.com/search?q="-Dtimingsafe_bcmp%3Dbcmp"</a><br />
<a href="https://www.google.com/search?q=%22%23define+timingsafe_bcmp+bcmp%22">https://www.google.com/search?q="%23define+timingsafe_bcmp+bcmp"</a>insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com20tag:blogger.com,1999:blog-833174317742362874.post-78284945931444708142014-04-29T16:15:00.000-07:002014-04-30T05:04:14.010-07:00GCC 4.9 Diagnostics<a href="http://gcc.gnu.org/">GCC</a> 4.9 <a href="http://gcc.gnu.org/gcc-4.9/changes.html">added support</a> for <a href="http://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Language-Independent-Options.html#index-fdiagnostics-color-252">colorizing</a> the warning and error output to make it easier to understand what went wrong during compilation of some source code. This probably wasn't added due to it being a good idea, but because <a href="http://clang.llvm.org/">clang</a> has this feature. I'm of the opinion that it's a good idea, but the GCC team probably isn't, as it's not turned on by default, and in order to use this feature you must specify some command line parameters or use some <a href="http://en.wikipedia.org/wiki/Environment_variable">environment variables</a>.<br />
<br />
The usage is described as follows:<code> </code><br />
<blockquote class="tr_bq">
<code>-fdiagnostics-color[=</code><var>WHEN</var><code>]</code><code></code><br />
<code>-fno-diagnostics-color</code><br />
<a href="https://www.blogger.com/blogger.g?blogID=833174317742362874" name="index-fdiagnostics-color-252"></a><a href="https://www.blogger.com/blogger.g?blogID=833174317742362874" name="index-highlight_002c-color_002c-colour-253"></a><a href="https://www.blogger.com/blogger.g?blogID=833174317742362874" name="index-GCC_005fCOLORS-_0040r_007benvironment-variable_007d-254"></a>Use color in diagnostics. <var>WHEN</var> is ‘<samp><span class="samp">never</span></samp>’, ‘<samp><span class="samp">always</span></samp>’,
or ‘<samp><span class="samp">auto</span></samp>’. <u>The default is ‘<samp><span class="samp">never</span></samp>’ if <samp><span class="env">GCC_COLORS</span></samp> environment
variable isn't present in the environment, and ‘<samp><span class="samp">auto</span></samp>’ otherwise</u>.
‘<samp><span class="samp">auto</span></samp>’ means to use color only when the standard error is a terminal.
The forms <samp><span class="option">-fdiagnostics-color</span></samp> and <samp><span class="option">-fno-diagnostics-color</span></samp> are
aliases for <samp><span class="option">-fdiagnostics-color=always</span></samp> and
<samp><span class="option">-fdiagnostics-color=never</span></samp>, respectively.</blockquote>
<blockquote class="tr_bq">
</blockquote>
So let's see how much GCC doesn't want you to add this feature. The default is to not show color. There's an option to forcefully turn off color<code> <i>-fno-diagnostics-color</i></code>. But wait, it gets even better. You read this paragraph and think to yourself: <i>"Hey, all I need to do is add <samp><span class="env">GCC_COLORS </span></samp>to my environment variables and I'll get color, right?"</i> But that's not the case either, as later on the documentation states:<br />
<blockquote class="tr_bq">
The default <samp><span class="env">GCC_COLORS</span></samp> is
‘<samp><span class="samp">error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01</span></samp>’
where ‘<samp><span class="samp">01;31</span></samp>’ is bold red, ‘<samp><span class="samp">01;35</span></samp>’ is bold magenta,
‘<samp><span class="samp">01;36</span></samp>’ is bold cyan, ‘<samp><span class="samp">01;32</span></samp>’ is bold green and
‘<samp><span class="samp">01</span></samp>’ is bold. <u>Setting <samp><span class="env">GCC_COLORS</span></samp> to the empty
string disables colors</u>.</blockquote>
Now let's ignore for the moment the complex notation needed for the environment string, and what it claims is the default setting of the environment variable if not specified. There's a logical discrepancy that setting an environment variable in your shell without any data is an empty string, which is then later buried as a final anecdote that an empty string in fact won't show any color!<br />
<br />
The best part about all this is that if you mistakenly set <samp><span class="env">GCC_COLORS</span></samp> without specifying any parameters, it overrides the command line parameters <code>-fdiagnostics-color=auto</code> and <code>-fdiagnostics-color=always</code>! Who cares what the documentation says, what sane behavior would be, or what you think all this means, nothing is more important than ensuring you do not have color!<br />
<br />
Now that we saw the lengths GCC goes to in order to ensure you won't use colors at all, let's compare the output to clang's.<br />
<br />
<div style="text-align: center;">
GCC:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-anAuTJAPOKM/U2AvT5SdOgI/AAAAAAAAALY/erxpbvh350s/s1600/gcc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-anAuTJAPOKM/U2AvT5SdOgI/AAAAAAAAALY/erxpbvh350s/s1600/gcc.png" height="117" width="320" /></a></div>
<br />
<div style="text-align: center;">
Clang:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-nnP83YOx6k4/U2AvZIGGPWI/AAAAAAAAALg/8exBdc1wBFw/s1600/clang.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-nnP83YOx6k4/U2AvZIGGPWI/AAAAAAAAALg/8exBdc1wBFw/s1600/clang.png" height="118" width="320" /></a></div>
<br />
The colors seem to be pretty much the same, and shows how hard GCC is trying to copy clang here. However, GCC some time back also added a caret <i>(^)</i> display to show what the warning is referring to, because clang has that. Yet as can be seen from these images, what clang is pointing to is actually useful, while GCC here rather unhelpfully points at the problematic functions, as opposed to what is actually wrong with it.<br />
<br />
GCC: 0.<br />
clang: +infinity.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com5tag:blogger.com,1999:blog-833174317742362874.post-65857948733993158872014-04-27T19:53:00.000-07:002014-04-30T10:11:52.183-07:00LibreSSL: The good and the bad<h3 style="text-align: center;">
OpenSSL & LibreSSL </h3>
<a href="http://www.openbsd.org/">OpenBSD</a> recently forked the popular <a href="http://en.wikipedia.org/wiki/Transport_Layer_Security">SSL/TLS</a> library <a href="http://www.openssl.org/">OpenSSL</a> into <a href="http://www.libressl.org/">LibreSSL</a>. Most of the reaction to this that I've seen tends to be pretty angry. People don't like the idea of a project being forked, they'd rather people work together, and have the OpenBSD team instead join OpenSSL.<br />
<br />
Now, for those of you that don't know it, OpenSSL is at the same time the best and most popular SSL/TLS library available, and <b>utter junk</b>. It's the best because it covers a wide array of the capabilities that exist across the many standards that make up SSL/TLS, and has seen years of development to iron out a multitude of issues and attacks levied against the specifications. The developers also seem to know a lot more about programming than the developers behind some of the competing SSL/TLS libraries. There are a ton of <a href="http://en.wikipedia.org/wiki/Gotcha_%28programming%29">gotchas</a> when it comes to developing an SSL/TLS stack, way beyond other areas of development, and things must be done absolutely correctly.<br />
<br />
<h3 style="text-align: center;">
Cryptographic development challenges</h3>
Aside from requiring meticulous programming to avoid typical programming issues and specific problems of the language in question, for SSL/TLS, one also needs to worry about:<br />
<ul>
<li>Ensuring libraries work correctly for tons of cases which are difficult to test for, and where unit tests cannot be exhaustive, and regression testing is far from complete.</li>
<li>Work in a completely hostile environment where every single outside variable must be validated by itself, and as part of a larger whole.</li>
<li>Data has to be correct both as raw data, and in their particular meaning.</li>
<li>Ensure data dependent operations run in actual <u>constant time</u> despite differences in their values to avoid <a href="http://en.wikipedia.org/wiki/Timing_attack">timing attacks</a>.</li>
<li>Ensure errors caught aren't actually handled immediately, but only at the very end of the single cryptographic unit as a whole completes, in order to avoid timing attacks. The rest of the unit must function normally despite the error(s).</li>
<li>Random data must be taken from an unpredictable source, and random streams must not have any detectable patterns or biases to them. </li>
<li>Sensitive/secret data must be protected, without allowing mathematical trickery or the computer environment somehow revealing them. </li>
<li>Ensuring data is wiped clean, without the <b>compiler optimizations</b> or virtual machine <i>ignoring</i> what they <u>deem to be pointless operations</u>.</li>
<li>The <b>inability to use some high-level languages</b> because they lack a way to tie in forceful cleanup of primitive data types, and their error handling mechanisms may end up leaving no way to wipe data, or data is duplicated without permission.</li>
<li>Almost every single thing which may be <u>the right way of doing things elsewhere is completely wrong where cryptography is concerned</u>.</li>
<li>Things still have to be extremely well optimized, otherwise the <b>expenses are too high to be viable in most scenarios</b>.</li>
</ul>
The above list is hardly exhaustive, and is pretty generic. Various cryptographic algorithms also have a ton of specific issues with them that need to be avoided. These issues are not necessarily specified in the standards, but have been learned over time from those that made mistakes, or by various research.<br />
<br />
Some examples of things to avoid which is part of the collective knowledge of SSL/TLS developers:<br />
<ul>
<li>Particular constants with certain algorithms must be avoided.</li>
<li>Particular constants with certain algorithms must be used. </li>
<li>Certain algorithms don't work well together.</li>
<li>Strings must also be handled as raw data.</li>
<li>Certain magnitudes are dangerous.</li>
<li>Some platforms process certain algorithms or data in a way which must be worked around.</li>
<li>Data broken up in certain ways is dangerous.</li>
</ul>
Now, one can read the specifications and implement them using best software engineering practices, and best cryptographic engineering practices, but where does one learn all the things to avoid? I've read 3 different books which cover how to implement significant parts or all of SSL/TLS, and <b>each one listed unique gotchas that weren't in the other two</b>. They're not mentioned in the various standards, and many of them are far from obvious. Unless you've spent years developing SSL/TLS, and took note of every mistake ever made, you're probably doing it wrong. In fact, most alternative SSL/TLS implementations I've seen make these various mistakes that OpenSSL already learned from.<br />
<br />
<h3 style="text-align: center;">
Who should design cryptographic libraries </h3>
<br />
In order to create a proper SSL/TLS implementation you need to be a master of:<br />
<ul>
<li>Cryptographic algorithms.</li>
<li>Cryptographic practice.</li>
<li>Software engineering.</li>
<li>Software optimization. </li>
<li>The language(s) used.</li>
<li>Domain specific knowledge.</li>
</ul>
Rarely are developers a true master at one of these, let alone being a master of the algorithms and software engineering. Which means most SSL/TLS libraries will either fail at being at the forefront of cryptography or fail nice sane design, or both.<br />
<br />
This is also why OpenSSL is utter junk. Some of its developers may have been decent at the algorithms and software engineering, but not particulary good at either of them. Other developers may have been a master of one, but absolutely abysmal at the other. Due to its popularity, <b>OpenSSL is also a dumping ground</b> for every new cryptographic idea, good or bad, and is constantly pushed in every direction, being spread far too thin.<br />
<br />
The OpenSSL API for most things is absolutely horrid. <u>No self-respecting software engineer could have designed them</u>. Objects which in reality share a sibling or parent-child relationship are implemented drastically differently, with dissimilar methods to work with them. Methods need to be used on objects in a certain unintuitive order for no apparent reason, or else things break.<br />
<br />
The source code under the hood is terrible too. A huge lack of consistency. Operations are done in a very roundabout manner. Issues with one platform are solved by making things worse for all other platforms. Most things are not implemented particularly well. In fact <b>the only thing OpenSSL is particularly good at is being ubiquitous</b>.<br />
<br />
<h3 style="text-align: center;">
Alternatives to OpenSSL</h3>
Now, several of the alternatives are much more nicely engineered and with saner APIs. However they all seem to fail basic quality in certain areas. For example, <a href="http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One">ASN.1</a> handling is generally written by people who don't know the ins and outs of <a href="http://en.wikipedia.org/wiki/Data_structure">data structures</a> and best practice and algorithms. <u>If you want good ASN.1, you'll need to have a database design expert to create it for you</u>, not the engineers who are better at cryptography or simple API design. It should also be repeated that <b>nice engineering doesn't equate secure</b>, there's just so much collective messy knowledge which needs to be added.<br />
<br />
So now enter the <i>OpenBSD developers</i>, who have a <b>track record for meticulous programming</b> and generally know what they're doing. They're looking to vastly cleanup OpenSSL. <u>I've been waiting for something like this for years</u>. They're going to take the best SSL/TLS library, and going to bring it up to decent engineering standards.<br />
<br />
Fixing OpenSSL itself with the OpenSSL team and development structure is really not an option. Since it's a dumping ground, there's no true quality control as needed. Since its main aim is to be ubiquitous, <u>the ancient platforms will inherently be dragging down the modern ones</u>.<br />
<br />
<h3 style="text-align: center;">
LibreSSL progress </h3>
So far, the OpenBSD fork of OpenSSL has deleted tons of code. This is crucial, as the more code there is, the more opportunity for bugs. Newbie programmers don't fully understand this point. Newbies think the more verbose some code is, the better. However, <b>more code means more room for mistakes</b>, and it also increases the code size. The more code used in a function, the harder it is to fully comprehend all of it. We <u>humans have limits to how much data we can juggle</u> in our heads. In order to ensure things can be fully conceptualized and analyzed, <b>it has to be as short and as modular as possible</b>. More library usage for similar techniques, and removing similar yet different copies of code littered throughout, as OpenBSD is aiming towards will ensure much higher levels of quality.<br />
<br />
OpenBSD is also working to ensure LibreSSL does not contain the <a href="http://en.wikipedia.org/wiki/Year_2038_problem">year 2038 problem</a>, extending compatibility far into the future. Some of the random method order usage requirements are removed, making <u>development less error prone with LibreSSL</u>. <i>LibreSSL</i> in a very short time is becoming much more lean and <b>more correct than OpenSSL ever was</b>.<br />
<br />
<h3 style="text-align: center;">
LibreSSL pitfalls </h3>
<br />
However, with the good, there is also bad. LibreSSL is aiming to be compatible with existing software using OpenSSL, which means <b>brain damaged APIs will continue to exist</b>. LibreSSL right now is also being modified to use OpenBSD specific functionality when OpenBSD's existing technology is more secure than what OpenSSL was doing. This means that for the time being, <u>LibreSSL will only work correctly with OpenBSD</u>. Since some functionality won't exist on other platforms, or that the functionality will exist, but not be nearly as secure as OpenBSD's implementation, potentially <b>making ports which seem straight forward to actually be less secure than OpenSSL currently is</b>.<br />
<br />
Once I saw what OpenBSD was doing, I predicted to myself that there would be those looking to port LibreSSL to other platforms, and fail to account for all the security considerations. It seems I was right. In just a few short days since LibreSSL's announcement, I'm seeing a multitude of porting projects pop up all over for Linux, Windows, OS X, FreeBSD, and more, all by developers who may know some software engineering, but <b>don't know the first thing about proper cryptographic implementations</b>, or understanding the specific advantages of the OpenBSD implementations.<br />
<br />
These porting implementations are failing many of the generic things listed above:<br />
<ul>
<li>Memory wiping is being optimized out.</li>
<li>Random values are not being seeded with proper <a href="http://en.wikipedia.org/wiki/Entropy_%28computing%29">entropy</a>.</li>
<li>Random values contain noticeable patterns or bias.</li>
<li><i>Bugs</i> fixed in OpenSSL by switching from generic memory functions to OpenBSD's <b>are being reintroduced</b> by being switched back to generic or naive implementations of the OpenBSD memory functions.</li>
<li>Constant-time functions aren't constant on the platforms being ported to.</li>
</ul>
The <u>developers not understanding what they're doing is why LibreSSL was created in the first place</u>. Unfortunately these porters are not security experts, and are creating or nabbing naive implementations from various sources. They're sometimes also grabbing source from OpenBSD itself or other security focused projects without realizing the source in question is being compiled with certain parameters or with special compilers to gain their security, and the implementation without the same build setup is hardly secure.<br />
<br />
As with OpenSSH, in the future, the OpenBSD team will probably be providing a portable LibreSSL. But until then, <b>avoid <a href="http://en.wikipedia.org/wiki/Counterfeit_consumer_goods">cheap knockoffs</a></b>.<br />
<br />
<h3 style="text-align: center;">
LibreSSL future</h3>
Will LibreSSL be successful? From what I've seen, it already is. You just may not have it available for your platform at the moment. <b>The API is still crud</b>, and as much as the OpenBSD team is cleaning things up, their code usually <u>could go a bit farther in terms of brevity and clarity</u>. However, the library will undoubtedly be <b>superior</b> to use compared to OpenSSL, and will serve as a <b>much cleaner starting point to document how to create other implementations</b>. Let's just hope the OpenBSD team is or will learn to be as knowledgeable regarding the SSL/TLS algorithms and gotchas as the OpenSSL team was.<br />
<br />
Now before you start clamoring to start creating a high level implementation of SSL/TLS, or complaining that C is bad, and it should be written in Java. Consider that <b>it's not even possible to program a secure SSL/TLS library in Java!</b> (Source: <a href="http://www.amazon.com/gp/product/0470474246/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470474246&linkCode=as2&tag=insacodi-20">Cryptography Engineering</a> page 122).insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com12tag:blogger.com,1999:blog-833174317742362874.post-55845685412207697792014-02-15T12:17:00.000-08:002014-02-15T14:54:10.597-08:00HTTP 308 Incompetence Expected<h3 style="text-align: center;">
Internet History</h3>
The Internet from every angle has always been a house of cards held together with defective duct tape. It's a miracle that anything works at all. Those who understand a lot of the technology involved generally hate it, but at the same time are astounded that for end users, things seem to usually work rather well.<br />
<br />
Today I'm going to point out some proposed changes being made to HTTP, the standard which the World Wide Web runs on. We'll see how not even the people behind the standards really know what they're doing anymore.<br />
<br />
The World Wide Web began in the early 90s in a state of flux. The Internet Engineering Task Force, as well as major players like Netscape released a bunch of quasi-standards to quickly build up a set of design rules and techniques used until HTTP v1.0 came out in 1995. Almost immediately after, HTTP v1.1 was being worked on, and despite not being standardized until 1999, it was pretty well supported in 1996. This is around the same time Internet Explorer started development, and a lot of their initial work was basically duplicating functionality and mechanics from Netscape Navigator.<br />
<br />
Despite standards and sane way of doing things, implementers always deviate from them, or come up with incorrect alternatives. Misunderstandings, and ideas on how things <i>should</i> work is how things were shaped in the early days.<br />
<br />
Thankfully though, over the years, standards online are finally being more strictly adhered to, and bugs are being fixed. Exact precise specifications exist for many things, as well as unit-tests to ensure adherence to standards. Things like Internet Explorer 6 are now a distant memory for most (unless you're in China). <br />
<br />
<h3 style="text-align: center;">
Existing Practice</h3>
A key point which led to many standards coming into existence was <i>existing practice</i>. Some browser or server would invent something, and the others would jump on board, and a standard would be created. Those who deviated were told to fix their implementation to match either the majority, or what was correct and would cause the least amount of issues for the long term stability of the World Wide Web.<br />
<br />
Now we'll see how today's engineers want to throw existing practice out the window, loosen up standards to the point of meaninglessness, and basically bust the technology you're currently using to view this article.<br />
<br />
<h3 style="text-align: center;">
HTTP Responses</h3>
One of the central designed structures of HTTP is that every response from a server has a code which identifies what the result is, and servers and clients should understand how to work with the particular responses. The more precise the definition, the better online experience we'll all have.<br />
<br />
<a href="http://www.w3.org/Protocols/HTTP/HTRESP.html" target="_blank">HTTP v0.9</a> was in a constant state of fluctuation, but offered three basic kinds of page redirects, permanent, temporary, and one which wasn't fully specified and unclear. These were defined as status codes 301, 302, and 303 respectively:<br />
<br />
<blockquote class="tr_bq">
Moved 301: The data requested has been assigned a new URI, the change is permanent.<br />
Found 302: The data requested actually resides under a different URL, however, the
redirection may be altered on occasion.<br />
Method 303: <i><b>Note: </b>This status code is to be specified in more detail. For
the moment it is for discussion only. </i>
<br />
Like the <i>found</i> response, this suggests that the client go try another network
address. In this case, a different <u>method</u> may be used. </blockquote>
<br />
The explanation behind a permanent and temporary redirect seems pretty straight forward. 303 is less clear, although it's the only one which mentions the <i>method</i> used is allowed to change, it's even the name associated with the response code.<br />
<br />
Several HTTP methods exist, for different kinds of activities. <i>GET</i> is a method to say, <i>hey, I want a page</i>. <i>POST</i> is a method to say, <i>hey here's some data from me, like my name and my credit card number, go do something with it</i>.<br />
<br />
The idea with the different redirects essentially was that 303 should embody <i>your requested was processed, please move on</i> (hence a POST request should now become a GET request), whereas 301 and 302 were to say what you need to do is elsewhere (permanently or temporarily), please take your business there (POST should remain POST).<br />
<br />
In any case, the text here was not as clear as can be, and developers were doing all kinds of things in general. <a href="http://tools.ietf.org/html/rfc1945#section-9.3" target="_blank">HTTP v1.0</a> came out to set the record straight.<br />
<br />
<blockquote class="tr_bq">
<pre class="newpage">301 Moved Permanently
The requested resource has been assigned a new permanent URL and
any future references to this resource should be done using that
URL. Clients with link editing capabilities should automatically
relink references to the Request-URI to the new reference returned
by the server, where possible. </pre>
</blockquote>
<blockquote class="tr_bq">
<pre class="newpage"> Note: When automatically redirecting a POST request after
receiving a 301 status code, some existing user agents will
<u>erroneously</u> change it into a GET request. </pre>
</blockquote>
<blockquote class="tr_bq">
<pre class="newpage">302 Moved Temporarily
The requested resource resides temporarily under a different URL.
Since the redirection may be altered on occasion, the client should
continue to use the Request-URI for future requests.</pre>
</blockquote>
<blockquote class="tr_bq">
<pre class="newpage"> Note: When automatically redirecting a POST request after
receiving a 302 status code, some existing user agents will
<u>erroneously</u> change it into a GET request. </pre>
</blockquote>
HTTP v1.0 however did not define 303 at all. Some developers not understanding what a temporary redirect is supposed to be thought it meant <i>hey, this is processed, now move on, however if you need something similar in the future, come here again</i>. We can hardly blame developers at that point for misusing 302, and wanting 303 semantics.<br />
<br />
<a href="http://tools.ietf.org/html/rfc2616#section-10.3.3" target="_blank">HTTP v1.1</a> decided to rectify this problem once and for all. 302 was renamed to <i>Found</i> and a new note was added:<br />
<i></i><br />
<pre class="newpage"> Note: <a href="http://tools.ietf.org/html/rfc1945">RFC 1945</a> and <a href="http://tools.ietf.org/html/rfc2068">RFC 2068</a> specify that the client is not allowed
to change the method on the redirected request. However, most
existing user agent implementations treat 302 as if it were a 303
response, performing a GET on the Location field-value regardless
of the original request method. The status codes 303 and 307 have
been added for servers that wish to make <b>unambiguously clear</b> which
kind of reaction is expected of the client.
</pre>
<br />
Since 302 was being used in two different ways, two new codes were created, one for each technique, to ensure proper use in the future. 302 retained its definition, but with so many incorrect implementations out there, 302 should essentially never be used if you want to ensure correct semantics are followed, instead use <i>303 - See Other</i><span class="h4"></span> (processing, move on...), or <i>307 Temporary Redirect</i><span class="h4"></span> (The real version of 302).<br />
<br />
In all my experience working with HTTP over the past decade, I've found 301, 303, and 307 to be implemented and used correctly as defined in HTTP v1.1, with 302 still being used incorrectly as 303 (instead of 307 semantics), generally by PHP programmers. But as above, never use 302, as who knows what the browser will do with it.<br />
<br />
Since existing practice today is that 301, 303, and 307 are used correctly pretty much everywhere, if someone misuses it, they should be told to correct their usage or handling. 302 is still so misused till this day, it's a lost cause.<br />
<br />
<h3 style="text-align: center;">
HTTP2 Responses</h3>
Now, in their infinite wisdom, the <a href="http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.4.2" target="_blank">new HTTP2 team has decided to create problems</a>. 301 status definition now brilliantly includes the following:<br />
<br />
<pre class="newpage"> Note: For historical reasons, a user agent MAY change the request
method from POST to GET for the subsequent request. If this
behavior is undesired, the 307 (Temporary Redirect) status code
can be used instead.</pre>
<br />
Let me get this straight, you're now taking a situation which hasn't been a problem for over a decade now, and asking it to begin happening anew by now allowing 301 to act as a 303???<br />
<br />
If you don't think that paragraph above was problematic, wait till you see <a href="http://tools.ietf.org/html/draft-reschke-http-status-308-07#section-1" target="_blank">this one</a>:<br />
<br />
<pre class="newpage"> +-------------------------------------------+-----------+-----------+
| | Permanent | Temporary |
+-------------------------------------------+-----------+-----------+
| Allows changing the request method from | 301 | 302 |
| POST to GET | | |
| Does not allow changing the request | - | 307 |
| method from POST to GET | | |
+-------------------------------------------+-----------+-----------+</pre>
<br />
301 is <i>allowed</i> to change the request method? Excuse me, I have to go vomit.<br />
<div style="text-align: center;">
<br /></div>
It was clear in the past that 301 was not allowed to change its method. But now, I don't even understand what this 301 is supposed to mean anymore. So I should permanently be using the new URI for GET requests. Where do my POSTs go? Are they processed? What the heck am I looking at?<br />
<br />
To add insult to injury, they're adding the new <i>308 Permanent Redirect</i><span class="h2"></span> as the <i>I really really mean I want true 301 semantics this time</i>. So now you can use a new status code which older browsers won't know what to do with, or the old status code that you're now allowing new browsers to utterly butcher for reasons I cannot fathom.<br />
<br />
Here's how the status codes work with HTTP 1.1:<br />
<br />
<pre class="newpage">+------+-------------------------------------+-----------+-----------------+
| Code | Meaning | Duration | Method Change |
+------+-------------------------------------+-----------+-----------------+
| 301 | Permanent Redirect. | Permanent | No |
| 302 | Temporary Redirect, misused often. | Temporary | Only by mistake |
| 303 | Process and move on. | Temporary | Yes |
| 307 | The true 302! | Temporary | No |
| 308 | Resume Incomplete, see below. | Temporary | No |
+------+-------------------------------------+-----------+-----------------+</pre>
<div>
<br /></div>
So here's how the status codes will work now with the HTTP2 updates:<br />
<br />
<pre class="newpage">+------+------------------------------+-----------+---------------+
| Code | Meaning | Duration | Method Change |
+------+------------------------------+-----------+---------------+
| 301 | Who the heck knows. | Permanent | Surprise Me |
| 302 | Who the heck knows. | Temporary | Surprise Me |
| 303 | Process and move on. | Temporary | Yes |
| 307 | The true 302! | Temporary | No |
| 308 | The true 301! | Permanent | No |
+------+------------------------------+-----------+---------------+</pre>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://cdn.memegenerator.net/instances/250x250/46071309.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br class="Apple-interchange-newline" /><img border="0" src="http://cdn.memegenerator.net/instances/250x250/46071309.jpg" /></a></div>
<div>
<br /></div>
And here's how one will have to do a permanent redirect in the future:<br />
<br />
<pre class="newpage"> +------+----------------+----------------+
| Code | Older Browsers | Newer Browsers |
+------+----------------+----------------+
| 301 | Correct. | Who Knows? |
| 308 | Broken!!! | Correct. |
+------+----------------+----------------+</pre>
<br />
This is how they want to alter things. Does this seem like a sane design to you?<br />
<br />
If the new design decisions of the HTTP2 team is to now capitulate to <b><u><i>rare</i></u></b> mistakes made out there, what's to stop here? I can see some newbie developers reading about how 307 and 308 are for redirects, misunderstanding them, and then misusing them too. So in five years we'll have 309 and 310 as <i>we really really really mean it this time</i>? This approach the HTTP2 team is taking is absurd. If you're going to invent new status codes each time you find an isolated instance of someone misusing one, where does it end?<br />
<br />
<h3 style="text-align: center;">
HTTP 308 is already taken!</h3>
One last point. Remember how earlier, I mentioned how a key point for the design of the Internet is to work with <i>existing practice</i>? 308 is in fact already used by something else, <a href="http://en.wikipedia.org/w/index.php?title=List_of_HTTP_status_codes&oldid=481535428#3xx_Redirection" target="_blank">Resume Incomplete</a> for resumable uploading. Which is <a href="https://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal" target="_blank">used by Google</a>, king of the Internet, and <a href="https://github.com/zendframework/zf2/pull/5013" target="_blank">many</a> <a href="http://tasvideos.org/EncoderGuidelines/TASVideosChannel.html#Steps" target="_blank">others</a>.<br />
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgur.com/34qNjgK.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://i.imgur.com/34qNjgK.jpg" height="282" width="320" /></a></div>
<div>
<br /></div>
<h3 style="text-align: center;">
Conclusion</h3>
I'm now dubbing HTTP 308 as <i>Incompetence Expected</i>, as that's clearly the only meaning it has. Or maybe that should be the official name for HTTP2 and the team behind it, I'll let you decide.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Edit:<br />
Thanks to those who read this article and sent in images. I added them where appropriate.<br />
<div>
<br /></div>
insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com20tag:blogger.com,1999:blog-833174317742362874.post-51542746170040247622013-04-02T18:09:00.000-07:002013-04-03T06:56:58.221-07:00Designing C++ functions to write/save to any storage mechanism<h3 style="text-align: center;">
Problem</h3>
A common issue when dealing with a custom object or any kind of data is to create some sort of save functionality with it, perhaps writing some text or binary to a file. So what is the correct C++ method to allow an object to save its data <u>anywhere</u>?<br />
<br />
An initial approach to allow some custom object to be able to save its data to a file is to create a member function like so:
<br />
<blockquote>
<pre>void save(const char *filename);</pre>
</blockquote>
While this is perfectly reasonable, what if I want something more advanced than that? Say I don't want the data to be saved as its own separate file, but would rather the data be written to some file that is already open, to a particular location within it? What if I'd rather save the data to a database? How about send the data over the network?<br />
<br />
<h3 style="text-align: center;">
Naive Approach</h3>
<br />
When C++ programmers hear the initial set of requirements, they generally look to one of two solutions:<br />
<br />
The first is to allow for a save function which can take an <i>std::ostream</i>, like so:
<br />
<blockquote>
<pre>void save(std::ostream &stream);</pre>
</blockquote>
C++ out of the box offers <i>std::cout</i> as an instance of an <i>std::ostream</i> which writes to the screen. C++ offers a derived class <i>std::ofstream</i> (<i>std::fstream</i>) which can save to files on disk. C++ also offers a derived class <a href="http://en.cppreference.com/w/cpp/io/basic_ostringstream" target="_blank">std::ostringstream</a> which saves file to a C++ string.<br />
<br />
With these options, you can display the data on the screen, save it to an actual file, or save it to a string, which you can then in turn save it wherever you want.<br />
<br />
The next option programmers look to is to overload <a href="http://en.cppreference.com/w/cpp/io/basic_ostream/operator_ltlt" target="_blank"><i>std::basic_ostream::operator<<</i></a> for the custom object. This way one can simply write:<br />
<blockquote>
<pre>mystream << myobject;</pre>
</blockquote>
And then the object can be written to any C++ stream.<br />
<br />
Either of these techniques pretty much work, but can be a bit annoying when you want a lot of flexibility <b>and</b> performance.<br />
<br />
Say I wanted to save my object over the network, what do I do? I could save it to a string stream, grab the string, and then send that over the network, even though that seems a bit wasteful.<br />
<br />
And for a similar case, say I have an already open <a href="http://en.wikipedia.org/wiki/File_descriptor" target="_blank">file descriptor</a>, and wish to save my object to it, do I also use a string stream as an intermediary?<br />
<br />
Since C++ is extensible, one could actually create their own <a href="http://en.cppreference.com/w/cpp/io/basic_streambuf" target="_blank">std::basic_streambuf</a> derived class which works with file descriptors, and attach it to an std::ostream, which can then be used with anything that works with a stream for output. I'm not going to go into the details how to do that here, but <a href="http://www.amazon.com/gp/product/0321623215/ref=as_li_qf_sp_asin_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321623215&linkCode=as2&tag=insacodi-20">The C++ Standard Library</a><img alt="" border="0" class="hioyqmdriwomkxxppfkw" height="1" src="http://www.assoc-amazon.com/e/ir?t=insacodi-20&l=as2&o=1&a=0321623215" style="border: none !important; margin: 0px !important;" width="1" /> explains the general idea, and provides a working <i>file descriptor streambuf</i> example and shows how to use it with stream functions. You can also find some ready made implementations online with a bit of searching, and some compilers may even include a solution out of the box in their C++ extensions.<br />
<br />
On UNIX systems, once you have a stream which works with file descriptors, you can now send data over the network, as sockets themselves are file descriptors. On Windows, you'll need a separate class which works with <i>SOCKET</i>s. Of course to turn a <i>file descriptor streambuf</i> into a <i>SOCKET</i> <i>streambuf</i> is trivial, and can probably be done with a few well crafted search and replace commands.<br />
<br />
Now this may have solved the extra string overhead with file descriptors and networking, but what about if I want to save to a database? What about if I'm working with C's <i>FILE *</i>? Does one now have to implement a new wrapper for each of these (or pray the compiler offers an extension, or one can be found online)? The C++ stream library is actually a bit bloaty, and creating your own streambufs is somewhat annoying, especially if you want to do it right and allow for buffering. Many stream related library code you find online are also of poor quality. Surely there must be a better option, right?<br />
<br />
<h3 style="text-align: center;">
Solution</h3>
<br />
If we look back at how C handles this problem, it uses <a href="http://insanecoding.blogspot.com/2007/05/secrets-to-optimization-function.html" target="_blank">function pointers</a>, where the function doing the writing receives a callback to use for the actual writing, and the programmer using it can make the writing go anywhere. C++ of course includes this ability, and even takes it much further, in the form of <a href="http://en.wikipedia.org/wiki/Function_object" target="_blank">function objects</a>, and even further in C++ 2011.<br />
<br />
Let's start with an example.<br />
<blockquote>
<pre>template<typename WriteFunction>
void world(WriteFunction func)
{
//Do some stuff...
//Do some more stuff...
func("World", 5); //Write 5 characters via callback
//Do some more stuff...
unsigned char *data = ...;
func(data, data_size); //Write some bytes
}
</pre>
</blockquote>
The template function above is expecting any function pointer which can be used to write data by passing it a pointer and a length. A proper signature would be something like the following:
<br />
<blockquote>
<pre>void func(const void *data, size_t length);</pre>
</blockquote>
Creating such a function is trivial. However, to be useful, writing needs to also include a destination of some sort, a device, a file, a database row, and so on, which makes function objects more powerful.<br />
<blockquote>
<pre>#include <cstdio>
class writer_file
{
std::FILE *handle;
public:
writer_file(std::FILE *handle) : handle(handle) {}
inline void operator()(const void *data, size_t length)
{
std::fwrite(data, 1, length, handle);
}
};
</pre>
</blockquote>
Which can be used as follows:
<br />
<blockquote>
<pre>world(writer_file(stdout));
</pre>
</blockquote>
Or perhaps:
<br />
<blockquote>
<pre>std::FILE *fp = fopen("somefile.bin", "wb");
world(writer_file(fp));
std::close(fp);
</pre>
</blockquote>
As can be seen, our World function can write to any <i>FILE *</i>.<br />
<br />
To allow any <i>char-based</i> stream to be written, the following function object will do the trick:<br />
<blockquote>
<pre>#include <ostream>
class writer_stream
{
std::ostream *handle;
public:
writer_stream(std::ostream &handle) : handle(&handle) {}
inline void operator()(const void *data, size_t length)
{
handle->write(reinterpret_cast<const char *>(data), length);
}
};
</pre>
</blockquote>
You can call this with:
<br />
<blockquote>
<pre>world(writer_stream(std::cout));</pre>
</blockquote>
Or anything in the <a href="http://www.cplusplus.com/reference/ostream/ostream/" target="_blank">ostream family</a>.<br />
<br />
If for some reason we wanted to write to strings, it's easy to create a function object for them too, and we can use the string directly without involving a string stream.<br />
<blockquote>
<pre>#include <string>
class writer_string
{
std::string *handle;
public:
writer_string(std::string &handle) : handle(&handle) {}
inline void operator()(const void *data, size_t length)
{
handle->append(reinterpret_cast<const char *>(data), length);
}
};
</pre>
</blockquote>
If you're worried about function objects being slow, then don't. Passing a function object like this to a template function has no overhead. The compiler is able to see a series of direct calls, and throws all the extraneous details away. It is as if the body of World is calling the write function to the handle passed to it directly. For more information, see <a href="http://www.amazon.com/gp/product/0201749629/ref=as_li_tf_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0201749629&linkCode=as2&tag=insacodi-20">Effective STL</a><img alt="" border="0" class="hioyqmdriwomkxxppfkw" height="1" src="http://www.assoc-amazon.com/e/ir?t=insacodi-20&l=as2&o=1&a=0201749629" style="border: none !important; margin: 0px !important;" width="1" /> Item 46.<br />
<br />
If you're wondering why developers forgo function pointers and function objects for situations like this, it is because
C++ offers so much with its stream classes, which are also very
extensible (and are often extended), they completely forget there are
other options. The stream classes are also designed for formatting output, and working with all kinds of special objects. But if you just need raw writing or saving of data, the stream classes are overkill.<br />
<br />
<h3 style="text-align: center;">
C++ 2011</h3>
<br />
Now C++ 2011 extends all this further in a multiple of ways.<br />
<br />
<h4 style="text-align: center;">
std::bind()</h4>
<br />
First of all, C++ 2011 offers <a href="http://en.cppreference.com/w/cpp/utility/functional/bind" target="_blank">std::bind()</a> which allows for creating function object adapters on the fly. std::bind() can take an unlimited amount of parameters. The first must be a function pointer of some sort, the next is optionally an object to work on in the case of a <a href="http://www.parashift.com/c++-faq/pointers-to-members.html" target="_blank">member function pointer</a>, followed by the parameters to the function. These parameters can be hard coded by the caller, or bound via placeholders by the callee.<br />
<br />
Here's how you would use std::bind() for using fwrite():<br />
<blockquote>
<pre>#include <functional>
world(std::bind(std::fwrite, std::placeholders::_1, 1, std::placeholders::_2, stdout));
</pre>
</blockquote>
Let us understand what is happening here. The function being called is std::fwrite(). It has 4 parameters. It's first parameter is the first parameter by the callee, denoted by <i>std::placeholders::_1</i>. The second parameter is being hard coded to 1 by the caller. The third parameter is the second parameter from the callee denoted by <i>std::placeholders::_2</i>. The fourth parameter is being hardcoded by the caller to stdout. It could be set to any <i>FILE *</i> as needed by the caller.<br />
<br />
Now we'll see how this works with objects. To use with a stream, the basic approach is as follows:<br />
<blockquote>
<pre>world(std::bind(&std::ostream::write, &std::cout, std::placeholders::_1, std::placeholders::_2));
</pre>
</blockquote>
Note how we're turning a member function into a pointer, and we're also turning <i>cout</i> into a pointer so it can be passed as <i>std::ostream::write</i>'s <i>this</i> pointer. The callee will pass its first and second parameters as the parameters to the stream write function.
However, the above has a slight flaw, it will only work if writing is done with <i>char *</i> data. We can solve that with casting.
<br />
<blockquote>
<pre>world(std::bind(reinterpret_cast<void (std::ostream::*)(const void *, size_t)>(&std::ostream::write), &std::cout, std::placeholders::_1, std::placeholders::_2));
</pre>
</blockquote>
Take a moment to notice that we're not just casting it to the needed function pointer, but as a member function pointer of <i>std::ostream</i>.<br />
<br />
You might find doing this a bit more comfortable than using classical function objects. However, function objects still have their place, wherever functions do. Remember, functions are about re-usability, and some scenarios are complicated enough that you want to pull out a full blown function object.<br />
<br />
For working with file descriptors, you might be tempted to do the following:<br />
<blockquote>
<pre>world(std::bind(::write, 1, std::placeholders::_1, std::placeholders::_2));</pre>
</blockquote>
This here will have World write to file descriptor 1 - generally standard output. However this simple design is a mistake. <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html">Write</a> can be interrupted by signals and needs to be resumed manually (by default, except on Solaris), among other issues, especially if the file descriptor is some kind of pipe or a socket. A proper write would be along the following lines:
<br />
<blockquote>
<pre>#include <system_error>
#include <unistd.h>
class writer_fd
{
int handle;
public:
writer_fd(int handle) : handle(handle) {}
inline void operator()(const void *data, size_t length)
{
while (length)
{
ssize_t r = ::write(handle, data, length);
if (r > 0) { data = static_cast<const char *>(data)+r; length -= r; }
else if (!r) { break; }
else if (errno != EINTR) { throw std::system_error(errno, std::system_category()); }
}
}
};</pre>
</blockquote>
<h4 style="text-align: center;">
Lambda Functions </h4>
Now you might be wondering, why C++ 2011 stopped with std::bind(), what if the function body needs more than just a single function call that can be wrapped up in an adapter? That's where lambda functions come in.
<br />
<blockquote>
<pre>world([&](const void *data, size_t length){ std::fwrite(data, 1, length, stdout); });
world([&](const void *data, size_t length){ std::cout.write(static_cast<const char *>(data), length); });
</pre>
</blockquote>
Note the ridiculous syntax. The <i>[](){}</i> combination signifies we are working with a lambda function. The <i>[]</i> receives a function scope, in this case <i>&</i>, which means that the function operates fully within its parent-scope, and has direct access to all its data. The rest you should already be well familiar with. You can change the <i>stdout</i> or the <i>cout</i> in the body of the lambda function to use your <i>FILE *</i> or <i>ostream</i> as necessary.<br />
<br />
Let us look at an example of having our World function write directly to a buffer.
<br />
<blockquote>
<pre>#include <cstring>
void *p = ...; //Point p at some buffer which has enough room to hold the contents needed to be written to it.
world([&](const void *data, size_t length){ std::memcpy(p, data, length); p = static_cast<char *>(p) + length; });</pre>
</blockquote>
There's a very important point in this example. There is a pointer which is initialized to where writing should begin. Every time data is written, the pointer is incremented. This ensures that if World calls the passed write function multiple times, it will continue to work correctly. This was not needed for files above, as their write pointer increments automatically, or with <i>std::string</i>, where <i>append</i> always writes to the end, wherever it now is.<br />
<br />
Be careful writing like this though, you must ensure in advance that your buffer is large enough, perhaps if your object has a way of reporting how much data the next call to its save or write function needs to generate. If it doesn't and you're winging it, something like the following is in order:<br />
<blockquote>
<pre>#include <stdexcept>
class writer_buffer
{
void *handle, *limit;
public:
writer_buffer(void *handle, size_t limit) : handle(handle), limit(static_cast<char>(handle)+limit) {}
inline void operator()(const void *data, size_t length)
{
if ((static_cast<char *>(handle) + length) > limit) { throw std::out_of_range("writer_buffer"); }
std::memcpy(handle, data, length);
handle = static_cast<char *>(handle) + length;
}
};
</char></pre>
</blockquote>
You can use it as follows:
<br />
<blockquote>
<pre>#include <cstdlib>
size_t amount = 1024; //A nice number!
void *buffer = std::malloc(amount);
world(writer_buffer(buffer, amount));</pre>
</blockquote>
Now an exception will be thrown if the callee tries to write more data than it should.<br />
<br />
<h4 style="text-align: center;">
std::function</h4>
Lastly, C++ 2011 added the ability for more verbose type checking on function objects, and the ability to create the save/write function as a normal function as opposed to a template function. That ability is a general reusable function object facade, <a href="http://en.cppreference.com/w/cpp/utility/functional/function" target="_blank">std::function</a>.<br />
<br />
To rewrite World to use it, we'd do as follows:
<br />
<blockquote>
<pre>void world(std::function<void (const void *, size_t)> func)
{
//Do some stuff...
//Do some more stuff...
func("World", 5); //Write 5 characters via callback
//Do some more stuff...
unsigned char *data = ...;
func(data, data_size); //Write some bytes
}
</pre>
</blockquote>
With <i>std::function</i>, the type is now made explicit instead of being a template. It is anything which receives any kind of buffer and its length, and returns nothing. This can ensure that callers will always use a compatible function as intended by the library designer. For example, in our case, the caller only needs to ensure that data can be passed via a <i>char *</i> and an <i>unsigned char *</i>, based on how World uses the callback function. If World was now modified to also output an <i>int *</i>, less capable callers would now break. std::function can ensure that things are designed properly up front. With <i>std::function</i>, you can now also restructure your code to place various components in different compilation units if you so desire, although perhaps at a performance penalty.<br />
<br />
<h3 style="text-align: center;">
Conclusion</h3>
To wrap up, you should now understand some features of C++ that are not as commonly used, or some new features of C++ 2011 that you may not be familiar with. You should now also have some ideas about generic code which should help you improve code you write.<br />
<br />
Many examples above were given only with one methodology, although they can be implemented with some of the others. For practice, try doing this yourself. Also try applying these ideas to other kinds of storage mechanisms not covered here, doing so should now be rather trivial for you.<br />
<br />
Remember, while this was done with a few standard examples and for writing, it can be extended to all handles Win32 offers, or for reading, or for anything else.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com13tag:blogger.com,1999:blog-833174317742362874.post-56079415696637885512013-03-19T05:12:00.000-07:002013-03-23T14:36:52.443-07:00OAuth - A great way to cripple your API<h3 style="text-align: center;">
Intro </h3>
A few years ago, the big social networking sites were looking for a secure way to allow their users to safely use any and all untrusted software to access their own personal accounts. So that a user could have their account on one social networking site safely interact with their account on another social networking site. They also wanted to allow for users to be able to allow their accounts to interact safely with various untrusted web applications and web sites.<br />
<br />
In order to safely allow untrusted software to access a user's account, a few points needed to be kept in mind. <br />
<ul>
<li>Untrusted software should not have access to a user's credentials, in case the software is compromised, passwords will not be stolen, as user's passwords are not stored by the software.</li>
<li>Untrusted software should not have full access to a user's account, but only limited access as defined by the user. In the same vein, giving the software user's personal credentials will allow unlimited access to a user's account, which could be used maliciously.</li>
<li>A user should be able to revoke the permission they granted to allow particular untrusted software to work, in case they turn malicious, despite the limited access they have.</li>
</ul>
A solution that was developed for this use-case was created - <a href="http://en.wikipedia.org/wiki/OAuth" target="_blank">OAuth</a>. OAuth's primary use-case is the one described above.<br />
<br />
<h3 style="text-align: center;">
Implementation </h3>
OAuth is generally implemented in a fashion where the untrusted software is accessed by a user via a standard web browser. In order for a user to authorize that software, the software will redirect the user's browser to a page on the social networking site, with a couple of parameters sent to it. Now that the user is on his social networking site, the untrusted software no longer has control over what the user is doing, as the user left the untrusted software, allowing the user to safely log in to his social networking account. Then, the social networking site will see the parameters it was passed, and ask the user if he or she wants to authorize the software in question, and what kind of access to the user's account it should be given.<br />
<br />
If and when the user has authorized the untrusted software, the social networking site can then report back to the untrusted software that it was granted a certain amount of access, and give it some credentials to use. These credentials are unique to the user and the software in question. The social networking site allows for a user to later on see a list of software authorized, and revoke the unique credentials given to any one of them, or modify the amount of access a particular set of credentials has. <br />
<br />
<h3 style="text-align: center;">
Problems</h3>
<h4 style="text-align: center;">
There is no standard</h4>
Now above, I said OAuth is <i>generally</i> implemented in this fashion. I say generally, because unlike standard HTTP authentication schemes (<a href="http://en.wikipedia.org/wiki/Basic_access_authentication" target="_blank">Basic</a>, <a href="http://en.wikipedia.org/wiki/Digest_access_authentication" target="_blank">Digest</a>, and others), OAuth is a big grab bag of ideas which can be mixed and matched in an infinite amount of ways, and also allows for developers to make their own unique tweaks to their personal implementations.<br />
<br />
With the standard HTTP authentication schemes, once a developer knows the protocol, and implemented it with one web site, that exact same knowledge can be reused to logging into any other web site that supports the standard. Likewise, software libraries can be made to handle HTTP authentication, and all a third party developer needs to do is specify to the library which credentials should be used, and then everything works as expected.<br />
<br />
With OAuth, once a developer learns how to authenticate with it to one web site, it is likely that the same developer will need to relearn how to connect to every other web site using OAuth. This further means that every web site which supports OAuth needs to document exactly how it is implementing it, and what tweaks are in use. It also means that no library can be written which can simply support every OAuth implementation out there. This places a great burden on developers on both sides. It can also greatly increase frustration for less able users, when their favorite library works great with one site they support, but are unable to extend their software to another site, because their library lacks some aspect of OAuth for this new site, or a unique OAuth tweak on this new site renders the library incompatible.<br />
<br />
Here are some choice quotes from the <a href="http://tools.ietf.org/html/rfc6749#section-1.8" target="_blank">official RFC</a>:<br />
<ul>
<li><i>However, as a rich and highly extensible framework with many optional components, on its own, this specification is likely to produce a wide range of <u>non-interoperable</u> implementations.</i></li>
<li><i>This framework was designed with the clear expectation that <u>future work</u> will define prescriptive profiles and extensions necessary to achieve full web-scale interoperability.</i></li>
</ul>
<br />
Another issue is that while various standard HTTP authentication schemes have well understood security margins, OAuth is entirely variable. On one web site, its OAuth implementation may be secure, while on another, <a href="http://www.thread-safe.com/2012/01/problem-with-oauth-for-authentication.html" target="_blank">its implementation may be Swiss cheese</a>. Even though a security consultant should generally look over any authorization implementation to ensure mistakes were not made, laymen can have a good understanding how reliable and secure their standardized authentication scheme is. A manager can ask their developers if some bullet points were adhered to, and then be reasonably confident in their security. Whereas with OAuth, the entire (complex) implementation needs to be reviewed from top to bottom by a top security professional to ensure it is secure. A manager has no adequate set of bullet points to discuss with their developers, as unique implementation details will drastically change the applicability of various points, with many important details still missing. At best, a manager can only get a false sense of security when OAuth is in use. The designers of OAuth respond to this point with a <b>71 page document</b> of <a href="http://tools.ietf.org/html/rfc6819" target="_blank">security issues</a> that need to be dealt with!<br />
<br />
What all this boils down to, is that OAuth is really just a set of ideas and recommendations on how to implement a unique authorization scheme. OAuth does not allow for interoperability as standards (usually) guarantee. Essentially, OAuth ensures that API authentication with your web site will be confusing, and will only work for the exact use-case for social networking sites described above.<br />
<br />
<h4 style="text-align: center;">
Crippling Design - APUI</h4>
The common OAuth use-case described above is to allow for a user on one web site to allow their software to communicate for them with another web site. The workflow described above requires that a user navigate between web sites using their browser and manually enter their credentials and authorization to various aspects of their account as part of the overall software authorization process.<br />
<br />
This means that OAuth (at least how it's normally implemented) only works between two web sites, with individual users, and that user intervention is required in order for software to authenticate with another web site. The fact that user intervention is required with manual input means that any API behind OAuth is not an <i>API</i> - Application Programming Interface, but an <b><i>AP<u>U</u>I</i></b> - <u>Application Programming <b>User</b> Interface</u>, meaning user intervention is required for functionality. All in all, this cripples your API, or APUI as it should now properly be called.<br />
<br />
Let us now focus on what cannot be properly done with OAuth in place:<br />
<ul>
<li>Third party software which is not part of a web site cannot interact with an OAuth <i>APUI</i>.</li>
<li>One user in third party software cannot act on behalf of another user via an OAuth <i>APUI</i>.</li>
<li>Third party software cannot run automated processes on an OAuth <i>APUI</i>.</li>
<li>Organizations cannot ensure tight integration between various software and services they use.</li>
</ul>
Let us review an example case where a company launches a new online service for coordinating schedules and personal calendars between people. We'll call this new online service <i>Calendar</i>. The developers of Calendar create an <i>APUI</i> for it using OAuth, so third party software can integrate with Calendar.<br />
<br />
As described above, a user needs to navigate between one web site and another in order to authorize software, with the two web sites sending each other information. What if one wants to integrate something which isn't a web site with Calendar? There's tons of desktop calendaring applications, Microsoft Outlook, Mozilla Lightning, Evolution, KOrganizer, Kontact, and more. If their developers want to integrate with the Calendar service, they need to embed a web browser in their applications now?<br />
<br />
Furthermore, that may not even help, if the OAuth workflow requires information be sent to an existing web site, what web site is associated with a user's personal copy of software? Developers of that software are unlikely to create a web site with user accounts just for this integration. Even if they did, if the site goes down (temporarily), then the integration stops working. Even though, in theory, there should be no dependance of some extra web site between Calendar and the software trying to integrate with it.<br />
<br />
<h4 style="text-align: center;">
Insecure </h4>
Also, as a security consideration, if an application does embed a web browser in it so it can authenticate with OAuth, the first requirement is no longer met - <i>Untrusted software should not have access to a user's credentials</i>. When using a standard web browser, once a user leaves the third party software's site and redirects to Calendar's site, the third party software cannot steal the information the user is entering. But when the third party software itself is the one which browses to Calendar's site, then it has access to everything the user is entering, including passwords.<br />
<br />
Actually, this attack works on OAuth in every circumstance, including web site to web site, and I've actually used it in practice. A web site can embed a web browser via a Java Applet or similar, or have a web browser server side which presents the OAuth log in page to the user, but slightly modified to have all the data entered pass through the third party site. Therefore OAuth doesn't even fulfill its own primary security objective!<br />
<br />
<h4 style="text-align: center;">
Incompatible with Enterprise</h4>
Next, once we go to the enterprise level, OAuth starts becoming much worse. Say a boss has his secretary manage his calendar for him, which is a very common scenario. In many OAuth setups, he cannot enter his Calendar credentials into the company-wide calendaring software running on a secure company server, which the secretary can then access. Rather, he would need to enter it directly in the browser the secretary uses, and stay there checking off many options. Also it is common that OAuth implementations are using security tokens which <i>expire</i>, meaning the boss will need to keep reentering his Calendar credentials again and again. Most bosses will just get fed up and give his secretary his credentials, especially if he's not physically near his secretary. It is also likely that the secretary will then write the password down. This gets compounded if multiple secretaries manage his schedule.<br />
<br />
Now say an organization has software for which it would like to run background processes for all its users with Calendar. Perhaps every week it would like to automatically analyze the Calendar accounts of all department chiefs to find a good time for a weekly meeting. How would this work exactly? Every department chief would now have to go and enter their credentials into this software? Then do it again each time security tokens expire from past authentications?<br />
<br />
<br />
Of course this is only the beginning. Since OAuth was designed for the likes of Twitter and Facebook which cater to individual personal accounts, the common implementations do not allow for hierarchical permissions as enterprise organizations need.<br />
<br />
In enterprise infrastructure, software is already used which has well defined roles and access rights for ever single user. This infrastructure should be deciding who can do what, and to what level one user can act on behalf of another user. Once an organization purchases accounts for all their employees from Calendar, no employee should be able to turn off or limit what the enterprise management software can do with their Calendar account. The enterprise management software is the one who needs to make such decisions. This flies in the face of <i>Untrusted software should not have full access to a user's account, but only limited access as defined by the user</i> and <i>A user should be able to revoke the permission they granted to allow particular untrusted software to work</i>.<br />
<br />
The amount of work involved to tightly integrate Calendar with existing infrastructure is also enormous from a user perspective. Every single account now has to have a user navigate across web pages and check all kinds of boxes until all the users are integrated. OAuth implementations generally forget to have administrator accounts which can do everything on behalf of other users in their organization.<br />
<br />
It should be obvious at this point whether enterprise organizations
would be willing to purchase accounts for their entire staff with
Calendar, when integration into their existing infrastructure or new
endeavors with the service will be difficult if not outright impossible.
Imagine how much money Calendar sales now stand to lose from being
unattractive to enterprise clients.<br />
<br />
<h4 style="text-align: center;">
Repel third party developers </h4>
OAuth implementations also generally require that any software which uses their <i>APUI</i>s be preregistered in advance. This places extra burden on third party developers. The OAuth implementations also commonly require that the third party web site integrating be preregistered in advance, so you can say goodbye to your application being compatible with staging servers which may be at another URL. It also makes software much less attractive when one party sells software to another party, as every new client now needs to go through an application registration process for the URLs they plan to setup the software they are purchasing. Therefore third party developers are less likely to be interested in selling software which integrates with Calendar, as there's hassle involved with every single sale.<br />
<br />
<h3 style="text-align: center;">
Recap </h3>
<ul>
<li>OAuth is not a standard, but a set of ideas, which does not allow third party developers to reuse knowledge, and places extra documentation burden on implementers.</li>
<li>OAuth as a whole has undefined security characteristics.</li>
<li>OAuth doesn't properly fulfill its primary security objectives.</li>
<li>OAuth doesn't work well outside social networking web site use-cases.</li>
<li>OAuth services are unattractive to enterprise organizations looking to integrate such services into their infrastructure.</li>
<li>OAuth services are less likely to have professional third parties sell software based upon them.</li>
</ul>
<br />
<br />
<h3 style="text-align: center;">
Solutions</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-nXwWoEBjKJg/UUg4VUOI83I/AAAAAAAAAJE/_yJpyAclQGQ/s1600/no-oauth.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-nXwWoEBjKJg/UUg4VUOI83I/AAAAAAAAAJE/_yJpyAclQGQ/s1600/no-oauth.png" /></a></div>
<br />
If you're looking to implement authorization for your API, I recommend to sticking with well understood secure designs, such as HTTP Basic Authentication over SSL/TLS (or HTTP Digest Authentication).<br />
<br />
In order to achieve a situation where users can securely authorize third party software, without giving over their personal credentials (passwords), I recommend that these services have a page where they can generate new credentials (keys) which the user can copy and paste. They can then name these keys themselves (avoiding application registration hassle), and set permissions upon them themselves. Since the user is the one initiating the key creation, and copying and pasting it themselves, they cannot fall prey to a <a href="http://insanecoding.blogspot.com/2009/11/malicious-hackers-are-not-out-there.html" target="_blank">man-in-the-middle attack</a> where the third party software initiates the authorization process.<br />
<br />
But remember the use-cases described here, and ensure that organizations have a way to access all user accounts company-wide, and without individual users being able to disable or limit that access.<br />
<br />
<h3 style="text-align: center;">
Conclusion</h3>
<br />
If you want your service to be used by everyone out there, be well supported by third parties, and to have them create all kinds of interesting software with it, <u>do <b>not</b> use OAuth</u>.<br />
<br />
Even the original social networking sites behind OAuth decided they really need other options for different use-cases, such as <a href="https://dev.twitter.com/docs/oauth/xauth" target="_blank">Twitter's xAuth,</a> or Yahoo offering <a href="http://developer.yahoo.com/messenger/guide/chapterdirectoauth.html" target="_blank"><i>Direct OAuth</i></a>, which turns the entire scheme into a more complicated version of HTTP Basic Authentication, with no added benefits. Perhaps the most damaging point against OAuth, is that the original designer behind it decided to <a href="http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/" target="_blank">remove his name from the specification, and is washing his hands clean of it</a>.<br />
<br />
I find it really amazing at how blind many big players are these days to all the problems with OAuth. When I first heard that IBM, a major enterprise player started offering services only accessible with typical utterly crippled OAuth, I was overcome with disbelief. Yet at the same time, I hear that they're <i>wondering</i> why they're not seeing the sales they used to with other services they offer, or compared to the competition.<br />
<br />
I'm even more amazed to see other big companies throwing away their currently working authentication systems for OAuth. Followed by them <i>wondering</i> why many third party developers are not upgrading to support the new authentication scheme, and clients jumping ship to inferior services.<br />
<br />
It seems many developers and managers out there simply <i>don't get it</i>. If you know anyone like that, show them this article.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com47tag:blogger.com,1999:blog-833174317742362874.post-1686930710281642662012-12-01T17:02:00.000-08:002012-12-01T17:09:08.665-08:00Debian breaks OSS4<p>Several people have contacted me to tell me that the latest version of OSS4 in Debian Unstable <i><a href="http://packages.debian.org/changelogs/pool/main/o/oss4/current/changelog" target="_blank">4.2-build2007-1+nmu1</a></i> introduces several audio issues such as garbled sound or kernel panics. I can confirm I have issues as well with this version.</p>
<p>Downgrading to <i>4.2-build2007-1</i> fixes the problem.</p>
<p>I recommending putting the following in <i>/etc/apt/preferences</i>:<br/>
<blockquote>
Package: oss4-base oss4-dev oss4-dkms oss4-gtk<br />Pin: version 4.2-build2007-1+nmu1<br />Pin-Priority: -1</blockquote>
This will prevent Debian from trying to upgrade to that version.
</p>
<p>If you already accidentally upgraded to it, older versions are still <a href="http://snapshot.debian.org/package/oss4/4.2-build2007-1/">available</a>.</p>insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com9tag:blogger.com,1999:blog-833174317742362874.post-24535638821056862732012-07-19T06:06:00.000-07:002012-07-23T07:46:56.132-07:00Creating portable Linux binariesFor some, the idea of creating a portable Linux binary is somewhat elusive.<br />
<br />
In this article, we will be discussing how to create a Linux binary for a specific architecture, that you will have great success running on a large variety of Linux distros. This includes current releases, somewhat old ones, and hopefully far into the future.<br />
<br />
A common problem facing those looking to deploy proprietary software on Linux, or for those trying to supply binaries to a very large user-base which will not compile your software themselves, is how to offer one binary that fits most normal scenarios.<br />
<br />
There are generally four naive approaches to solving this problem.<br />
<br />
<ol>
<li>The developers set up a bunch of specific distros, and compile the software on each of them, and give out distro specific binaries. This makes sense at first, till you run into some trouble. <ul>
<li>You have to juggle many live CDs or maintain a bunch of installed distros which is painful and time consuming.</li>
<li>You end up only offering support for the distros you have handy, and you will get quite a few users on a more exotic distro nagging you for support, or a different and incompatible version of a distro you're already supporting.</li>
<li>The compilers or other build utilities on some distros are too old for your modern software, and you need to build them elsewhere, or figure out how to back port modern software to that old distro.</li>
<li>Builds break when a user upgrades their system.</li>
<li>Users end up needing to install some non standard system libraries, increasing everyone's frustration.</li>
</ul>
</li>
<li>The developers just statically link the binaries. This isn't always a legal option due to some licenses that may be involved. Binaries which are fully statically linked also in many instances exhibit incorrect behavior (more on this later). </li>
<li>The developers just compile the software on one system, and pray that it works for everyone else.</li>
<li>Compile with a really really old distro and hope it works everywhere else. However this succumbs to the last three problems outlined in naive approach #1, and in many cases the binaries produced won't work with modern distros.</li>
</ol>
Now there are plenty of companies that supply those portable Linux binaries. You find on their website downloads for say Linux i386, AMD64, and PPC. Somehow that i386 binary manages to run on every i386 system you've tested, Red Hat, Debian, SUSE, Ubuntu, Gentoo, and both old and modern versions at that. What is their secret sauce?<br />
<br />
Now let us dive into all the important information and techniques to accomplish this worthy goal.<br />
<br />
First thing you want to know is what exactly is your binary linked to anyway? For this, the handy <i>ldd</i> command comes in.<br />
<br />
<blockquote>
/tmp> ldd myapp<br />
linux-vdso.so.1 => (0x00007fff7a1ff000)<br />
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1f8a765000)<br />
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1f8a4e3000)<br />
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1f8a2cc000)<br />
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1f89f45000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00007f1f8aaa9000)<br />
/tmp> </blockquote>
The lines which are directed to a file are system libraries that you need to worry about. In this case, there's libstdc++, libm, libgcc, and libc. libc and libm are both part of (<a href="http://www.eglibc.org/home" target="_blank">E</a>)<a href="http://www.gnu.org/software/libc/" target="_blank">GLIBC</a>, the C library that most Linux applications will be using. <a href="http://gcc.gnu.org/libstdc++/" target="_blank">libstdc++</a> is <a href="http://gcc.gnu.org/" target="_blank">GCC</a>'s C++ library. <a href="http://gcc.gnu.org/onlinedocs/gccint/Libgcc.html" target="_blank">libgcc</a> is GCC's implementation of some programming constructs that your program may be using, such as exception handling, and things like that.<br />
<br />
In general (E)GLIBC is broken up into many sub libraries that your program may be linked against. Other notable examples are libdl for Dynamic Loading, libpthread for threading, librt for various real time functions, and a few others.<br />
<br />
Your application will not run on a system unless all these dependencies are found, and are compatible. Therefore, versions of these also come into play. In general a newer minor version of a library will work, but not an older.<br />
<br />
In order to find versions numbers, you want to use <i>objdump</i>. Here's an example with finding out what version of (E)GLIBC is needed:<br />
<blockquote>
/tmp> objdump -T myapp | grep GLIBC_<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 ungetc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3 __ctype_toupper_loc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fputc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 free<br />
…<br />
/tmp></blockquote>
In this case, 2.3 is the highest version number. Therefore this binary needs (E)GLIBC 2.3 or higher on the system. Note, the version numbers have nothing to do with the version installed on your system, rather (E)GLIBC marks each function with the minimum version that contains it.<br />
<br />
Of course all this applies to other libraries as well, particularly libgcc and libstdc++.<br />
<br />
Now that we know a little bit about what we're doing, I'm going to present the first bit of secret sauce.<br />
<br />
If you're using C++, link with <i>-static-libstdc++</i> this will ensure that libstdc++ is linked statically, but won't link every other lib statically like <i>-static</i> would. You want libstdc++ linked statically, because it's safe to do so, some systems may be using an older version (or have none at all, in the case of some servers), or you want your binary to remain compatible if a new major version of libstdc++ comes out which is no longer backwards compatible. Note that even though libstdc++ is GPL'd, it also offers a <a href="http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html" target="_blank">linking exception</a> that allows you to link against it and even statically link it in any application.<br />
<br />
If you see that your binary needs libgcc, also use <i>-static-libgcc</i> for the same reasons given above. Also, GCC is GPL'd, and has the same <a href="http://www.gnu.org/licenses/gcc-exception.html" target="_blank">linking exception</a> as above. I once had the unfortunate scenario where I sold a client an application without libgcc statically linked, that used exceptions. On his old server, as long as everything went absolutely perfectly, the application ran fine, but if any issue occurred, instead of gracefully handling the issue, the application terminated immediately. Since his libgcc was too old, the application saw the throws, but none of the catches. Statically linking libgcc fixed this issue.<br />
<br />
Now you might be thinking, hey what about statically linking (E)GLIBC? Let me warn you that doing so is a bad idea. Some features in (E)GLIBC will only work if the statically linked (E)GLIBC is the exact same version of (E)GLIBC installed on the system, making statically linking pointless, if not downright problematic. (E)GLIBC's libdl is quite notable in this regard, as well as several networking functions. (E)GLIBC is also licensed under <a href="http://www.gnu.org/software/libc/manual/html_node/Copying.html#Copying" target="_blank">LGPL</a>. Which essentially means that if you give out the source to your application, then in most cases you can distribute statically linked binaries with it, but otherwise, not. Also, since 99% of the functions are marked as requiring extremely old versions of (E)GLIBC, statically linking is hardly necessary in most cases.<br />
<br />
The next bit of the secret sauce is statically linking those non standard libs your application needs but nothing else.<br />
<br />
You probably never learned in school how to selectively static link those libraries you want, but it is indeed possible. Before the list of libraries you wish to static link, place <i>-Wl,-Bstatic</i> and afterwards <i>-Wl,-Bdynamic</i>.<br />
<br />
Say in my application I want to statically link <a href="http://curl.haxx.se/" target="_blank">libcurl</a> and <a href="http://www.openssl.org/" target="_blank">OpenSSL</a>, but want to dynamically link <a href="http://zlib.net/" target="_blank">zlib</a>, and the rest of my libs, such as other parts of (E)GLIBC, I would use the following as my link flags:<br />
<i>gcc -o app *.o <u>-static-libgcc -Wl,-Bstatic -lcurl -lssl -lcrypto -Wl,-Bdynamic</u> -lz -ldl -lpthread -lrt </i><br />
<br />
The next step is to ensure that your libraries pull in as few dependencies as possible. Here's the output from <i>ldd</i> on my libcurl.so:<br />
<blockquote>
linux-vdso.so.1 => (0x00007fffbadff000)<br />
libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f84410a4000)<br />
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f8440e7b000)<br />
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f8440c6b000)<br />
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f8440a1a000)<br />
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8440812000)<br />
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f84405d2000)<br />
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8440374000)<br />
libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f843ff90000)<br />
librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f843fd75000)<br />
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f843fb5e000)<br />
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f843f7d7000)<br />
libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f843f558000)<br />
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f843f342000)<br />
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f843f127000)<br />
libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f843ee66000)<br />
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f843ec4a000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00007f844157e000)<br />
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f843e976000)<br />
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f843e74c000)<br />
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f843e548000)<br />
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f843e33f000)<br />
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f843e13a000)<br />
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f843df36000)<br />
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f843dd32000)<br />
libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f843db21000)<br />
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f843d90f000)</blockquote>
This is quite unacceptable. Distros generally compile packages with everything enabled. Your application generally does not need everything a library has to offer. In the case of libcurl, you can compile it yourself, and disable the features you aren't using. In an example application, I only need HTTP and FTP support, so I could compile libcurl with very little, and now have this:<br />
<blockquote>
linux-vdso.so.1 => (0x00007fffbadff000)<br />
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8440812000)<br />
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8440374000)<br />
libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f843ff90000)<br />
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f843fb5e000)<br />
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f843f7d7000)<br />
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f843f342000)<br />
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f843ec4a000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00007f844157e000)<br />
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f843e13a000) </blockquote>
This is much more manageable. Refer to the documentation of your libraries in order to see how to compile them without features you don't need.<br />
<br />
If you're going to be compiling your own libraries, you probably want to set up a second system, virtual machine, or a <a href="http://en.wikipedia.org/wiki/Chroot" target="_blank">chroot</a> for building your customized library versions and applications, to ensure it doesn't conflict with your main system. Especially for the upcoming tip.<br />
<br />
Secret sauce part 3, push your (E)GLIBC requirements down.<br />
<br />
Let's look at an <i>objdump</i> on OpenSSL.<br />
<blockquote>
/tmp> objdump -T /usr/lib/libcrypto.so.0.9.8 | grep GLIBC_<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 chmod<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fileno<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __sysv_signal<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 printf<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memset<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 ftell<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getgid<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 shutdown<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 close<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 syslog<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 ioctl<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 abort<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memchr<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 gethostbyname<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fseek<br />
<b>0000000000000000 DF *UND* 0000000000000000 GLIBC_2.7 __isoc99_sscanf</b><br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 openlog<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 exit<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strcasecmp<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 gettimeofday<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 setvbuf<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 read<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strncmp<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 malloc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fopen<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 setsockopt<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 sysconf<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getpid<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fgets<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 geteuid<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 vfprintf<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 closelog<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fputc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 times<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 free<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strlen<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 ferror<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 opendir<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __xstat<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 listen<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3 __ctype_b_loc<br />
0000000000000000 w DF *UND* 0000000000000000 GLIBC_2.2.5 __cxa_finalize<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 readdir<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 dlerror<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 sprintf<br />
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.2.5 stdin<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strrchr<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 dlclose<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 poll<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getegid<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 gmtime_r<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strerror<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 sigaction<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strcat<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getsockopt<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fputs<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 lseek<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strtol<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getsockname<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 connect<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memcpy<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memmove<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strchr<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 socket<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fread<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __fxstat<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getenv<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __errno_location<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 qsort<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strncasecmp<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strcmp<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strcpy<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getuid<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3 __ctype_tolower_loc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memcmp<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 feof<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fclose<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 dlopen<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 recvfrom<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strncpy<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 dlsym<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 closedir<br />
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.2.5 stderr<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fopen64<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 sendto<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 bind<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fwrite<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 realloc<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 perror<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fprintf<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 localtime<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 write<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 accept<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strtoul<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 open<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 time<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fflush<br />
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getservbyname<br />
/tmp></blockquote>
It seems that OpenSSL would work on (E)GLIBC 2.3, except for one pesky function which needs 2.7+. This is a problem if I want to ship an application with this modern OpenSSL on say Red Hat Enterprise Linux 5 which comes with GLIBC 2.5, or say Debian Stable from ~4 years ago, which only has 2.4.<br />
<br />
In this case OpenSSL is using a C99 version of sscanf(), but not actually by choice.<br />
<br />
In <i>/usr/include/stdio.h</i> on (E)GLIBC 2.7+, you'll notice two blocks:<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#if defined __USE_ISOC99 && !defined __USE_GNU \</span>
<span style="color: green;"> && (!defined __LDBL_COMPAT || !defined __REDIRECT) \</span>
<span style="color: green;"> && (defined __STRICT_ANSI__ || defined __USE_XOPEN2K)</span>
<span style="color: green;"># ifdef __REDIRECT</span>
<span style="color: grey; font-style: italic;">/* For strict ISO C99 or POSIX compliance disallow %as, %aS and %a[</span>
<span style="color: grey; font-style: italic;"> GNU extension which conflicts with valid %a followed by letter</span>
<span style="color: grey; font-style: italic;"> s, S or [. */</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT (fscanf, (FILE *__restrict __stream,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...),</span>
<span style="color: black;"> __isoc99_fscanf) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT (scanf, (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...),</span>
<span style="color: black;"> __isoc99_scanf) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT_NTH (sscanf, (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...),</span>
<span style="color: black;"> __isoc99_sscanf);</span>
<span style="color: green;"># else</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_fscanf (FILE *__restrict __stream,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_scanf (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_sscanf (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, ...) __THROW;</span>
<span style="color: green;"># define fscanf __isoc99_fscanf</span>
<span style="color: green;"># define scanf __isoc99_scanf</span>
<span style="color: green;"># define sscanf __isoc99_sscanf</span>
<span style="color: green;"># endif</span>
<span style="color: green;">#endif</span></pre>
<br />
And<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;"># if !defined __USE_GNU \</span>
<span style="color: green;"> && (!defined __LDBL_COMPAT || !defined __REDIRECT) \</span>
<span style="color: green;"> && (defined __STRICT_ANSI__ || defined __USE_XOPEN2K)</span>
<span style="color: green;"># ifdef __REDIRECT</span>
<span style="color: grey; font-style: italic;">/* For strict ISO C99 or POSIX compliance disallow %as, %aS and %a[</span>
<span style="color: grey; font-style: italic;"> GNU extension which conflicts with valid %a followed by letter</span>
<span style="color: grey; font-style: italic;"> s, S or [. */</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT (vfscanf,</span>
<span style="color: black;"> (FILE *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format, _G_va_list __arg),</span>
<span style="color: black;"> __isoc99_vfscanf)</span>
<span style="color: black;"> __attribute__ ((__format__ (__scanf__, </span><span style="color: blue;">2</span><span style="color: black;">, </span><span style="color: blue;">0</span><span style="color: black;">))) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT (vscanf, (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format,</span>
<span style="color: black;"> _G_va_list __arg), __isoc99_vscanf)</span>
<span style="color: black;"> __attribute__ ((__format__ (__scanf__, </span><span style="color: blue;">1</span><span style="color: black;">, </span><span style="color: blue;">0</span><span style="color: black;">))) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __REDIRECT_NTH (vsscanf,</span>
<span style="color: black;"> (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format,</span>
<span style="color: black;"> _G_va_list __arg), __isoc99_vsscanf)</span>
<span style="color: black;"> __attribute__ ((__format__ (__scanf__, </span><span style="color: blue;">2</span><span style="color: black;">, </span><span style="color: blue;">0</span><span style="color: black;">)));</span>
<span style="color: green;"># else</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_vfscanf (FILE *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format,</span>
<span style="color: black;"> _G_va_list __arg) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_vscanf (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format,</span>
<span style="color: black;"> _G_va_list __arg) __wur;</span>
<span style="color: black; font-weight: bold;">extern</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> __isoc99_vsscanf (__const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __s,</span>
<span style="color: black;"> __const </span><span style="color: maroon;">char</span><span style="color: black;"> *__restrict __format,</span>
<span style="color: black;"> _G_va_list __arg) __THROW;</span>
<span style="color: green;"># define vfscanf __isoc99_vfscanf</span>
<span style="color: green;"># define vscanf __isoc99_vscanf</span>
<span style="color: green;"># define vsscanf __isoc99_vsscanf</span>
<span style="color: green;"># endif</span>
<span style="color: green;"># endif</span></pre>
<br />
These two blocks of code make fscanf(), scanf(), sscanf(), vfscanf(), vscanf(), and vsscanf() use special C99 versions. Since older applications were already compiled against C89 versions, (E)GLIBC doesn't want to potentially break them and change how an existing function works. So instead, a new set of functions were created which only exist in (E)GLIBC 2.7+, and (E)GLIBC by default will direct all calls to these functions to the proper C99 versions when compiling.<br />
<br />
Now there are some defines you can set in your library code and application code to ensure it uses the old more backwards compatible versions, but getting the exact right combination of defines without breaking anything else can be tricky. It may also be tedious to modify a code-base you're not familiar with.<br />
<br />
Therefore, I recommend just deleting these two blocks from your <i><stdio.h></i> on your build system. You want your build system to be able to build everything for backwards compatibility, right?<br />
<br />
If you're recompiling libraries like OpenSSL which are designed for massive portability with all kinds of systems, odds are, they're not looking for C99 support in basic scanf() family functions anyway. If you do happen to need C99 scanf() support in your application, I recommend that you add it manually with a specialized lib, for maximum portability. You can easily <a href="http://daniel.haxx.se/projects/trio/" target="_blank">find a bunch online</a>.<br />
<br />
The last scenario that you may encounter is that you happen to want to use a modern library function. For most libs you can just statically link them, but that won't work for (E)GLIBC. Since some functions depend on system support, or that custom versions don't perform as well as the built in system ones, you definitely want to use the built in ones if they're available. The question is, how to once the binary has already been compiled?<br />
<br />
So for our final bit of secret sauce, dynamically load any modern functions that you want to use, and work around them, or disable some functionality if not present.<br />
<br />
Remember libdl that we mentioned above? It offers <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html" target="_blank">dlopen()</a> for opening system libraries, and <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlsym.html" target="_blank">dlsym()</a> for finding out if certain functions are present or not, and retrieving a pointer to them.<br />
<br />
I'm going to post a full example that you can look at and play with. In this example, we have a program which tries to figure out how big system <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/pipe.html" target="_blank">pipes</a> are. In this application, we are going to see how much data we can stuff in a pipe before
we're told that the pipe is full, and the write would need to block.<br />
<br />
Linux offers a function called <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/pipe.2.html" target="_blank">pipe2()</a> which has the crucial ability to create a pipe in non-blocking mode. If it doesn't exist, we can create it ourselves, but we prefer the built in one if possible.<br />
<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#ifndef __linux__</span>
<span style="color: green;">#error This program is specifically designed for Linux, even though it works elsewhere</span>
<span style="color: green;">#endif</span>
<span style="color: green;">#include <stdio.h> //puts(), fputs(), printf(), fprintf(), stderr</span>
<span style="color: green;">#include <errno.h> //errno, perror(), EINTR, EAGAIN, EWOULDBLOCK</span>
<span style="color: green;">#include <fcntl.h> //fcntl(), F_SETFL, F_GETFL, O_NONBLOCK, F_SETFD, F_GETFD, FD_CLOEXEC, and for some: O_CLOEXEC</span>
<span style="color: green;">#include <dlfcn.h> //dlopen(), dlsym(), dlclose(), dlerror(), RTLD_LAZY</span>
<span style="color: green;">#include <unistd.h> //pipe() used in our implementation, write(), close()</span>
<span style="color: grey; font-style: italic;">//Lifted from: /usr/include/<arch>/bits/fcntl.h</span>
<span style="color: green;">#ifndef O_CLOEXEC</span>
<span style="color: green;">#define O_CLOEXEC 02000000 </span><span style="color: grey; font-style: italic;">/* set close_on_exec */</span>
<span style="color: green;">#endif</span>
<span style="color: grey; font-style: italic;">//End lift</span>
<span style="color: black; font-weight: bold;">typedef</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> (*pipe2_t)(</span><span style="color: maroon;">int</span><span style="color: black;"> [</span><span style="color: blue;">2</span><span style="color: black;">], </span><span style="color: maroon;">int</span><span style="color: black;">);</span>
<span style="color: grey; font-style: italic;">//Implement the rather straight forward pipe2(), note: this function is of type: static pipe2_t</span>
<span style="color: maroon;">static</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> our_pipe2(</span><span style="color: maroon;">int</span><span style="color: black;"> pipefd[</span><span style="color: blue;">2</span><span style="color: black;">], </span><span style="color: maroon;">int</span><span style="color: black;"> flags)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> ret = pipe(pipefd);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!ret) </span><span style="color: grey; font-style: italic;">//Success, pipe created</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//The built in pipe2() would not suffer from race conditions that the following code would succumb to in a threaded application</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (flags & O_NONBLOCK)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> fcntl(pipefd[</span><span style="color: blue;">0</span><span style="color: black;">], F_SETFL, fcntl(pipefd[</span><span style="color: blue;">0</span><span style="color: black;">], F_GETFL) | O_NONBLOCK);</span>
<span style="color: black;"> fcntl(pipefd[</span><span style="color: blue;">1</span><span style="color: black;">], F_SETFL, fcntl(pipefd[</span><span style="color: blue;">1</span><span style="color: black;">], F_GETFL) | O_NONBLOCK);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (flags & O_CLOEXEC)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> fcntl(pipefd[</span><span style="color: blue;">0</span><span style="color: black;">], F_SETFD, fcntl(pipefd[</span><span style="color: blue;">0</span><span style="color: black;">], F_GETFD) | FD_CLOEXEC);</span>
<span style="color: black;"> fcntl(pipefd[</span><span style="color: blue;">1</span><span style="color: black;">], F_SETFD, fcntl(pipefd[</span><span style="color: blue;">1</span><span style="color: black;">], F_GETFD) | FD_CLOEXEC);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(ret);</span>
<span style="color: black;">}</span>
<span style="color: maroon;">static</span><span style="color: black;"> pipe2_t pipe2 = our_pipe2; </span><span style="color: grey; font-style: italic;">//pipe2() is initialized to our function</span>
<span style="color: black;">size_t pipe_size() </span><span style="color: grey; font-style: italic;">//Manually determine the size of the system's pipe, for automatic, look up Linux specific F_GETPIPE_SZ</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Create a union for using a pipe, so usage is a bit more logical</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">union</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> pipefd[</span><span style="color: blue;">2</span><span style="color: black;">];</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">struct</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> read;</span>
<span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> write;</span>
<span style="color: black;"> } side;</span>
<span style="color: black;"> } u;</span>
<span style="color: black;"> size_t amount = </span><span style="color: blue;">0</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//A pipe size of 0 signifies unknown</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (!pipe2(u.pipefd, O_NONBLOCK)) </span><span style="color: grey; font-style: italic;">//Note, here pipe2() is used</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">for</span><span style="color: black;"> (;;) </span><span style="color: grey; font-style: italic;">//Write to a pipe in a loop, the final amount should be the size of the pipe</span>
<span style="color: black;"> {</span>
<span style="color: black;"> ssize_t w = write(u.side.write, &amount, </span><span style="color: black; font-weight: bold;">sizeof</span><span style="color: black;">(size_t)); </span><span style="color: grey; font-style: italic;">//Write a size_t to the pipe</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (w > </span><span style="color: blue;">0</span><span style="color: black;">) { amount += w; } </span><span style="color: grey; font-style: italic;">//Success, add amount written and then loop</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (w == </span><span style="color: blue;">0</span><span style="color: black;">) </span><span style="color: grey; font-style: italic;">//Pipe was closed, and we certainly didn't close it</span>
<span style="color: black;"> {</span>
<span style="color: black;"> perror(</span><span style="color: #dd0000;">"Pipe unexpectedly closed"</span><span style="color: black;">);</span>
<span style="color: black;"> amount = </span><span style="color: blue;">0</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Reset to unknown, because an error occured</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">break</span><span style="color: black;">;</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> </span><span style="color: grey; font-style: italic;">/* Error occured trying to write */</span><span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (errno != EINTR) </span><span style="color: grey; font-style: italic;">//And it wasn't an interruption, so something that needs handling</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> ((errno != EAGAIN) && (errno != EWOULDBLOCK)) </span><span style="color: grey; font-style: italic;">//Failed to write to pipe - and it's nothing we'd fix</span>
<span style="color: black;"> {</span>
<span style="color: black;"> perror(</span><span style="color: #dd0000;">"Failed to write to pipe"</span><span style="color: black;">);</span>
<span style="color: black;"> amount = </span><span style="color: blue;">0</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//Reset to unknown, because an error occured</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Else, pipe is full, we're done!</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">break</span><span style="color: black;">; </span><span style="color: grey; font-style: italic;">//In either case, we're done writing to the pipe</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Else If (errno == EINTR), we'd just loop and try again</span>
<span style="color: black;"> }</span>
<span style="color: black;"> close(u.side.read);</span>
<span style="color: black;"> close(u.side.write);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> { perror(</span><span style="color: #dd0000;">"Failed to create pipe"</span><span style="color: black;">); }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(amount);</span>
<span style="color: black;">}</span>
<span style="color: maroon;">int</span><span style="color: black;"> main(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">int</span><span style="color: black;"> argc, </span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *</span><span style="color: maroon;">const</span><span style="color: black;"> *</span><span style="color: maroon;">const</span><span style="color: black;"> argv)</span>
<span style="color: black;">{</span>
<span style="color: black;"> </span><span style="color: maroon;">void</span><span style="color: black;"> *so = dlopen(</span><span style="color: #dd0000;">"libc.so.6"</span><span style="color: black;">, RTLD_LAZY); </span><span style="color: grey; font-style: italic;">//Open the C library</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (so)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: maroon;">void</span><span style="color: black;"> *sym = dlsym(so, </span><span style="color: #dd0000;">"pipe2"</span><span style="color: black;">); </span><span style="color: grey; font-style: italic;">//Grab the handle to pipe2() if it exists</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (sym) </span><span style="color: grey; font-style: italic;">//Success!</span>
<span style="color: black;"> {</span>
<span style="color: black;"> pipe2 = (pipe2_t)sym; </span><span style="color: grey; font-style: italic;">//Use the built in one instead of ours</span>
<span style="color: black;"> puts(</span><span style="color: #dd0000;">"Using system's pipe2()."</span><span style="color: black;">);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> { puts(</span><span style="color: #dd0000;">"Using our pipe2()."</span><span style="color: black;">); }</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span>
<span style="color: black;"> {</span>
<span style="color: black;"> puts(</span><span style="color: #dd0000;">"Using our pipe2()."</span><span style="color: black;">);</span>
<span style="color: black;"> fprintf(stderr, </span><span style="color: #dd0000;">"Could not open C library: %s</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, dlerror());</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: grey; font-style: italic;">//Here's the real work</span>
<span style="color: black;"> size_t a = pipe_size();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (a) { printf(</span><span style="color: #dd0000;">"Pipe size is: %zu</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, a); }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">else</span><span style="color: black;"> { fputs(</span><span style="color: #dd0000;">"Could not determine pipe size.</span><span style="color: magenta;">\n</span><span style="color: #dd0000;">"</span><span style="color: black;">, stderr); }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (so) { dlclose(so); }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(</span><span style="color: blue;">0</span><span style="color: black;">);</span>
<span style="color: black;">}</span></pre>
Here's how to compile and run it:
<br />
<blockquote>
/tmp> gcc -Wall -o pipe_test pipe_test.c -ldl<br />
/tmp> ./pipe_test<br />
Using system's pipe2().<br />
Pipe size is: 65536<br />
/tmp></blockquote>
Now pipe2() Was added to GLIBC in 2.9, yet this binary here according to <i>objdump</i> only needs (E)GLIBC 2.2.5+. Here's the output from an older system with GLIBC 2.7, using the exact same binary created on a newer system:
<br />
<blockquote>
/tmp> ./pipe_test<br />
Using our pipe2().<br />
Pipe size is: 65536<br />
/tmp></blockquote>
Lastly, let me recap all the techniques we learned.
<br />
<ul>
<li>Use <i>ldd</i> and <i>objdump</i> to see version requirements of binaries and libraries. </li>
<li>Statically link compiler and language libraries, such as libgcc and libstdc++.</li>
<li>Statically link selected libraries, while dynamically linking others.</li>
<li>Compile selected libraries with as little needed functionality as possible. </li>
<li>Pushing (E)GLIBC requirements down, by being wary of functions which have changed over time, and (E)GLIBC redirects calls to them in newly compiled programs by default.</li>
<li>Pushing (E)GLIBC requirements down by not directly using new functions, and instead working around their presence.</li>
</ul>
Doing all this, you'll still need to make different builds for different operating systems, and different architectures like x86 and ARM, but at least you won't be forced to for all different distros and versions thereof.<br />
<br />
One thing of note, it's possible to have Linux with different C libraries, and in those cases, you may as well be using a different Operating System. You'll be hard pressed to make complex programs compiled against one C library run on Linux which uses a different C library, where the needed one is not present. Thankfully though, all the mainstream desktop and server distros all use (E)GLIBC.<br />
<br />
In any case, the techniques you've learned here can also be applied to other setups too. (E)GLIBC was only focused on in this article because of its popularity and its many gotchas, but many other libraries that you may use, particularly video and audio libraries have similar issues as well.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com12tag:blogger.com,1999:blog-833174317742362874.post-57329906156069838332011-12-14T07:58:00.000-08:002011-12-14T14:01:25.135-08:00Progression and Regression of Desktop User InterfacesAs this Gregorian year comes to a close, with various new interfaces out now, and some new ones on the horizon, I decided to recap my personal experiences with user interfaces on the desktop, and see what the future will bring.<br /><br />When I was younger, there were a few desktop environments floating around, and I've seen a couple of them at school or a friend's house. But the first one I had on my own personal computer, and really played around with was Windows 3.<br /><br />Windows 3 came with something called Program Manager. Here's what it looked like:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/-LGd15rvRKqA/TujJXGRNAKI/AAAAAAAAAHk/c26JCwBHpDk/s1600/Program_Manager.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 210px;" src="http://4.bp.blogspot.com/-LGd15rvRKqA/TujJXGRNAKI/AAAAAAAAAHk/c26JCwBHpDk/s320/Program_Manager.png" alt="" id="BLOGGER_PHOTO_ID_5686015928212193442" border="0" /></a><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/-F9Y_kTh71Bw/TujJXdQsy0I/AAAAAAAAAHs/Z3yG7hGFsb8/s1600/program-manager-internet-explorer.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="http://4.bp.blogspot.com/-F9Y_kTh71Bw/TujJXdQsy0I/AAAAAAAAAHs/Z3yG7hGFsb8/s320/program-manager-internet-explorer.png" alt="" id="BLOGGER_PHOTO_ID_5686015934384098114" border="0" /></a><br /><br />The basic idea was that you had a "start screen", where you had your various applications grouped by their general category. Within each of these "groups", you had shortcuts to the specific applications. Now certain apps like a calculator you only used in a small window, but most serious apps were only used maximized. If you wanted to switch to another running app, you either pressed the alt+tab keyboard shortcut to cycle through them, or you minimized everything, where you then saw a screen listing all the currently running applications.<br /><br />Microsoft also shipped a very nice file manager, known as "File Manager", which alone made it useful to use Windows. It was rather primitive though, and various companies released various add-ons for it to greatly enhance its abilities. I particularly loved Wiz File Manager Pro.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/-f3kvvBB5TGI/TujmZowSikI/AAAAAAAAAH8/0nTN6sTK34c/s1600/fileman.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="http://3.bp.blogspot.com/-f3kvvBB5TGI/TujmZowSikI/AAAAAAAAAH8/0nTN6sTK34c/s320/fileman.png" alt="" id="BLOGGER_PHOTO_ID_5686047857666329154" border="0" /></a><br /><br />Various companies also made add-ons for Program Manager, such as to allow nested groups, or shortcuts directly on the main screen outside of a group, or the ability to change the icons of selected groups. Microsoft would've probably built some of these add-ons in if it continued development of Program Manager.<br /><br />Now not everyone used Windows all the time back then, but only selectively started it up when they wanted something from it. I personally did everything in DOS unless I wanted to use a particular Windows app, such as file management or painting, or copying and pasting stuff around. Using Windows all the time could be annoying as it slowed down some DOS apps, or made some of them not start at all due to lack of memory and other issues.<br /><br />In the summer of 1995, Microsoft released Windows 4 to the world. It came bundled with MS-DOS 7, and provided a whole new user experience.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/-VDrDcboMELI/Tujo1PoXc-I/AAAAAAAAAII/BEtbuoVwtcg/s1600/windows95.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="http://2.bp.blogspot.com/-VDrDcboMELI/Tujo1PoXc-I/AAAAAAAAAII/BEtbuoVwtcg/s320/windows95.png" alt="" id="BLOGGER_PHOTO_ID_5686050530981802978" border="0" /></a><br /><br />Now the back-most window, the "desktop", no longer contained a list of the running programs (in explorer.exe mode), but rather you could put your own shortcuts and groups and groups of groups there. Rather running programs would appear at the bottom of the screen in a "taskbar". The taskbar now also contained a clock, and a "start menu", to launch applications. Some always running applications which were meant to be background tasks appeared as a tiny icon next to the clock in an area known as a "tray".<br /><br />This was a major step forward in usability. Now, no matter which application you were currently using, you could see all the ones that were currently running on the edge of your screen. You could also easily click on one of them to switch to it. You didn't need to minimize all to see them anymore. Thanks to the start menu, you could also easily launch all your existing programs without needing to minimize all back down to Program Manager. The clock always being visible was also a nice touch.<br /><br />Now when this came out, I could appreciate these improvements, but at the same time, I also hated it. A lot of us were using 640x480 resolutions on 14" screens back then. Having something steal screen space was extremely annoying. Also with how little memory systems had back at the time (4 or 8 MB of RAM), you generally weren't running more than 2 or 3 applications at a time and could not really appreciate the benefits of having an always present taskbar. Some people played with taskbar auto-hide because of this.<br /><br />The start menu was also a tad ridiculous. Lots of clicks were needed to get anywhere. The default appearance also had too many useless things on it.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/-qMXIhbbGRWM/TujrotSaWaI/AAAAAAAAAIU/BPeKanOiPag/s1600/startmenu.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 320px;" src="http://3.bp.blogspot.com/-qMXIhbbGRWM/TujrotSaWaI/AAAAAAAAAIU/BPeKanOiPag/s320/startmenu.png" alt="" id="BLOGGER_PHOTO_ID_5686053614139365794" border="0" /></a><br />Did anyone actually use help? Did most people launch things from documents? Microsoft released a nice collection of utilities called "PowerToys" which contained "TweakUI" which you could use to make things work closer to how you want.<br /><br />The default group programs installed to within the start menu was quite messy though. Software installers would not organize their shortcuts into the groups that Windows came with, but each installed their apps into their own unique group. Having 50 submenus pop out was rather unwieldy, and I personally organized each app after I installed it. Grouping into "System Tools", "Games", "Internet Related Applications", and so on. It was annoying to manually do all this though, as when removing an app, you had to remove its group manually. On upgrades, one would also have to readjust things each time too.<br /><br />Windows 4 also came with the well known Windows Explorer file manager to replace the older one. It was across the board better than the vanilla version of File Manager that shipped with Windows 3.<br /><br />I personally started dual booting MS-DOS 6.22 + Windows for Workgroups 3.11 and Windows 95 (and later tri-booted with OS/2 Warp). Basically I used DOS and Windows for pretty much everything, and Windows 95 for those apps that required it. Although I managed to get most apps to work with Windows 3 using Win32s.<br /><br />As I got a larger screen and more RAM though, I finally started to appreciate what Windows 4 offered, and started to use it almost exclusively. I still exited Windows into DOS 7 though for games that needed to use more RAM, or ran quicker that way on our dinky processors from back then.<br /><br />Then Windows 4.10 / Internet Explorer 4 came out which offered a couple of improvements. First was "quick launch" which allowed you to put shortcuts directly on your taskbar. You could also make more than one taskbar and put all your shortcuts on it. I personally loved this feature, I put one taskbar on top of my screen, and loaded it with shortcuts to all my common applications, and had one on the bottom for classical use. Now I only had to dive into the start menu for applications I rarely used.<br /><br />It also offered a feature called "active desktop" which made the background of the desktop not just an image, but a web page. I initially loved the feature, as I edited my own page, and stuck in an input line which I would use to launch a web browser to my favorite search engine at the time (which changed monthly) with my text already searched for. After a while active desktop got annoying though, as every time IE crashed, it disabled it, and you had to deal with extra error messages, and go turn it on manually.<br /><br />By default this new version also made every Windows Explorer window have this huge sidebar stealing your precious screen space. Thankfully though, you could turn it off.<br /><br />All in all though, as our CPUs got faster, RAM became cheaper, and large screens more available, this interface was simply fantastic. I stopped booting into other OSs, or exiting Windows itself.<br /><br />Then Windows 5 came out for consumers, and UI wise, there weren't really any significant changes. The default look used these oversized bars and buttons on each window, but one could easily turn that off. The start menu got a bit of a rearrangement to now feature your most used programs up front, and various actions got pushed off to the side. Since I already put my most used programs on my quick launch on top, this start menu was a complete waste of space. Thankfully, it could also be turned off. I still kept using Windows 98 though, as I didn't see any need for this new Windows XP, and it was just a memory hog in comparison at the time.<br /><br />What was more interesting to me however was that at work, all our machines ran Linux with GNOME and KDE. When I first started working there, they made me take a course on how to use Emacs, as every programmer needs a text editor. I was greatly annoyed by the thing however, where was my shift highlight with shift+delete and shift+insert or ctrl+x and ctrl+v cut and paste? Thankfully though I soon found gedit and kedit which was like edit/notepad/wordpad but for Linux.<br /><br />Now I personally don't use a lot of windowsy type software often. My primary usage of a desktop consists of using a text editor, calculator, file manager, console/terminal/prompt, hex editor, paint, cd/dvd burner, and web browser. Only rarely do I launch anything else. Perhaps a PDF, CHM, or DJVU reader when I need to read something.<br /><br />After using Linux with GNOME/KDE at work for a while, I started to bring more of my work home with me and wanted these things installed on my own personal computer. So dual booting Windows 98 + Linux was the way to go. I started trying to tweak my desktop a bit, and found that KDE was way more capable than GNOME, as were most of their similar apps that I was using. KDE basically offered me everything that I loved about Windows 98, but on steroids. KWrite/KATE was notepad/wordpad but on steroids. The syntax highlighting was absolutely superb. KCalc was a fine calc.exe replacement. Konqueror made Windows Explorer seem like a joke in comparison.<br /><br />Konqueror offered network transparency, thumbnails of everything rather quickly, even of text files (no pesky thumbs.db either!). An info list view which was downright amazing:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/-IoKmHGtj0HE/Tuj4EwBsoeI/AAAAAAAAAIg/tFdhJ1qhkcM/s1600/konq.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 130px;" src="http://2.bp.blogspot.com/-IoKmHGtj0HE/Tuj4EwBsoeI/AAAAAAAAAIg/tFdhJ1qhkcM/s320/konq.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5686067290050437602" /></a>This is a must have feature here. Too often are documents distributed under meaningless file names. With this, and many other small features, nothing else I've seen has even come close to Konqueror in terms of file management.<br /><br />Konsole was just like my handy DOS Prompt, except with tab support, and better maximizing, and copy and paste support. KHexEdit was simply better than anything I had for free on Windows. KolourPaint is mspaint.exe with support for way more image formats. K3b was also head and shoulders above Easy CD Creator or Nero Burning ROM. For Web Browsers, I was already using Mozilla anyway on Windows, and had the same option on Linux too.<br /><br />For the basic UI, not only did KDE have everything I liked about Windows, it came with an organized start menu. Which also popped out directly, instead from a "programs" submenu.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/-5r8L9sci1ds/Tuj5oNzFjuI/AAAAAAAAAIs/ojcvSTR0NJY/s1600/kdestart.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 229px; height: 191px;" src="http://4.bp.blogspot.com/-5r8L9sci1ds/Tuj5oNzFjuI/AAAAAAAAAIs/ojcvSTR0NJY/s320/kdestart.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5686068998849269474" /></a><br />The taskbar was also enhanced that I could stick various applets on it. I could stick volume control <span style="font-weight:bold;">directly</span> on it. Not a button which popped out a slider, the sliders themselves could appear on the taskbar. Now for example, I could easily adjust my microphone volume directly, without popping up or clicking on anything extra. There was an eyedropper which you could just push to find out the HTML color of anything appearing on the screen - great for web developers. Another thing which I absolutely love, I can see all my removable devices listed directly on my taskbar. If I have a disk in my drive, I see an icon for that drive appearing directly on my taskbar, and I can browse it, burn to it, eject it, whatever. With this, everything I need is basically at my finger tips.<br /><br />Before long I found myself using Linux/KDE all the time. On newer machines I started to dual boot Windows XP with Linux/KDE, so I could play the occasional Windows game when I wanted to, but for real work, I'd be using Linux.<br /><br />Then KDE 4 comes out, and basically half the stuff I loved about KDE was removed. No longer is it Windows on steroids. Now KDE 4.0 was not intended for public consumption. Somehow all the distros except for Debian seemed to miss that. Everyone wants to blame KDE for miscommunicating this, but it's quite clear 4.0 was only for developers if you watched the KDE 4 release videos. Any responsible package maintainer with a brain in their skull should've also realized that 4.0 was not ready for prime time. Yet it seems the people managing most distros are idiots that just need the "latest" version of everything, ignoring if it's better or not, or even stable.<br /><br />At the time, all these users were upset, and all started switching to GNOME. I don't know why anyone who truly loved the power KDE gave you would do that. If KDE 3 > GNOME 2 > KDE 4, why did people migrate to GNOME 2 when they could just not "upgrade" from KDE 3? It seems to me that people never really appreciated what KDE offers in the first place if they bothered moving to GNOME instead of keeping what was awesome.<br /><br />Nowadays people tell me that KDE 4 has feature parity with KDE 3, but I have no idea what they're talking about. The Konqueror info list feature that I described above still doesn't seem to exist in KDE 4. You can no longer have applets directly on your taskbar. Now I have to click a button to pop up a list of my devices, and only then can I do something with them. No way to quickly glance to see what is currently plugged in. Konsole's tabs now stretch to the full width of your screen for some pointless reason. If you want to switch between tabs with your mouse, prepare for carpal tunnel syndrome. Who thought that icons should grow if they can? It's exactly like those idiotic theoretical professors who spout that CPU usage must be maximized at all times, and therefore use algorithms that schedule things for maximum power draw, despite that in normal cases performance does not improve by using these algorithms. I'd rather pack in more data if possible, having multiple columns of information instead of just huge icons.<br /><br />KHexEdit has also been completely destroyed. No longer is the column count locked to hex (16). I can't imagine anyone who seriously uses a hex editor designed the new version. For some reason, KDE now also has to act like your mouse only has one button, and right click context menus vanished from all over the place. It's like the entire KDE development team got invaded by Mac and GNOME users who are too stupid to deal with anything other than a big button in the middle of the screen.<br /><br />Over in the Windows world. Windows 6 (with 6.1 comically being consumerized as "Windows 7") came out with a bit of a revamp. The new start menu seems to fail basic quality assurance tests for anything other than default settings. Try to set things to offer a classic start menu, this is what you get:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/-Mt2z6cUg-dQ/Tuj-8q2QzqI/AAAAAAAAAI4/IV8Nn-Le6PQ/s1600/windows7.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 267px; height: 320px;" src="http://1.bp.blogspot.com/-Mt2z6cUg-dQ/Tuj-8q2QzqI/AAAAAAAAAI4/IV8Nn-Le6PQ/s320/windows7.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5686074847802740386" /></a><br /><br />If you use extra large icons for your grandparents, you also find that half of the Control Panel is now inaccessible. Ever since Bill Gates left, it seems Windows is going down the drain.<br /><br />But hardly are problems solely for KDE and Windows, GNOME 3 is also a major step back according to what most people tell me. Many of these users are now migrating to XFCE. If you like GNOME 2, why are you migrating to something else for? And what is it with people trying to fix what isn't broken? If you want to offer an alternate interface, great, but why break or remove the one you already have?<br /><br />Now a new version of Windows is coming out with a new interface being called "Metro". They should really be calling it "Retro". It's Windows 3 Program Manager with a bunch of those third party add-ons, with a more modern look to it. Gone is the Windows 4+ taskbar so you can see what was running, and easily switch applications via mouse. Now you'll need to press the equivalent of a minimize all to get to the actual desktop. Another type of minimize to get back to Program Manager to launch something else, or "start screen" as they're now calling it.<br /><br />So say goodbye to all the usability and productivity advantages Windows 4 offered us, they want to take us back to the "dark ages" of modern computing. Sure a taskbar-less interface makes sense on handheld devices with tiny screens or low resolution, but on modern 19"+ screens? The old Windows desktop+taskbar in the upcoming version of Windows is now just another app in their Metro UI. So "Metro" apps won't appear on the classic taskbar, and "classic" applications won't appear on the Metro desktop where running applications are listed.<br /><br />I'm amazed at how self destructive the entire market became over the last few years. I'm not even sure who to blame, but someone has to do something about it. It's nice to see that a small group of people took KDE 3.5, and are continuing to develop it, but they're rebranding everything with crosses everywhere and calling it "Trinity" desktop. Just what we need, to bring religious issues now into desktop environments. What next? The political desktop?insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com22tag:blogger.com,1999:blog-833174317742362874.post-19341689966936217182011-11-29T12:26:00.001-08:002011-11-29T16:50:28.509-08:00Reading in an entire file at once in C++, part 2Last week I discussed <a href="http://insanecoding.blogspot.com/2011/11/how-to-read-in-file-in-c.html">6 different methods on how to quickly get an entire file into a C++ string</a>.<br /><br />The conclusion was that the straight forward method was the ideal method, this was verified across several compilers. Since then, people asked me a number of questions.<br /><br />What if I want to use a vector instead of a string, are the speeds different?<br />Forget C++ containers, what about directly into a manually allocated buffer?<br />What about copying into a buffer via mmap?<br />What do these various cases say about compilers or their libraries? Can this indicate what they're good or bad at compiling?<br /><br />So to establish some baseline numbers, I created an application which reads the same files the same amount of times using the fastest method possible. It directly creates a buffer aligned to the partition's blocking factor, informs the OS to do the fastest possible sequential read, and then reads the file directly into the buffer. I also tested the suggested method to mmap a file, and copy the contents from there to an allocated buffer. The times achieved for these were 5 and 9 respectively. The latter is slower because of the extra memcpy() required, you want to read directly into your destination buffer whenever possible. The fastest time I now achieved should more or less be the fastest theoretical limit I'm able to get with my hardware. Both GCC and LLVM got the exact same times. I did not test with VC++ here as it doesn't support POSIX 2008.<br /><br />Now regarding our 6 methods from last time, all of them except the Rdbuf method are possible with std::vector, since there is no std::vector based std::istream.<br /><br />An interesting thing to note is that C++ string implementations vary greatly from implementation to implementation. Some offer optimizations for very small strings, some offer optimizations for frequent copies, by using reference counting. Some always ensure the string ends with a 0 byte so you can immediately pass them as a C string. In this latter case, operations which operate on strings as a range are rather quick, as the data is copied, then a 0 is appended. Whereas a loop which constantly pushes bytes on the back will have to needlessly set the extra trailing byte to 0 each time. Vector implementations on the other hand don't need to worry about a trailing 0 byte, and generally don't try to internally use all kinds of elaborate storage methods. So if std::vector works for you, you may want to use that.<br /><br />Let's review times for the 5 applicable methods with our various compilers.<br /><br />GCC 4.6 with a vector:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>23.5</td></tr><tr><td>C++</td><td>22.8</td></tr><tr><td>Iterator</td><td>73</td></tr><tr><td>Assign</td><td>81.8</td></tr><tr><td>Copy</td><td>68</td></tr></table><br /><br />Whereas with a string:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>24.5</td></tr><tr><td>C++</td><td>24.5</td></tr><tr><td>Iterator</td><td>64.5</td></tr><tr><td>Assign</td><td>68</td></tr><tr><td>Copy</td><td>63</td></tr></table><br /><br />We see that with a vector, the basic methods became a bit faster, but interestingly enough, the others got slower. However, which methods are superior to the others have remained the same.<br /><br />Now for LLVM 3 with a vector:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>8</td></tr><tr><td>C++</td><td>8</td></tr><tr><td>Iterator</td><td>860</td></tr><tr><td>Assign</td><td>1328</td></tr><tr><td>Copy</td><td>930</td></tr></table><br /><br />Versus for string:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>7.5</td></tr><tr><td>C++</td><td>7.5</td></tr><tr><td>Iterator</td><td>110</td></tr><tr><td>Assign</td><td>102</td></tr><tr><td>Copy</td><td>97</td></tr></table><br /><br />With LLVM, everything is slower with a vector, and for the more complex solutions, much much slower. There's two interesting things we can see about LLVM though. For more straight forward logic, their compiler's optimizations are extremely smart. The speeds approach the theoretical best. I did some profiling on GCC and LLVM, as they're using the same C and C++ libraries, and found that in the straight C/C++ methods for my test program, GCC made 300 memory allocations, but LLVM made only 200. LLVM apparently is smart enough to see inside the various allocations, skip the ones that aren't needed, and place the data directly into the output buffer. But for complex code, LLVM's optimizations aren't that great. In the case of vectors and iterators, downright awful. Someone should file some bug reports with them.<br /><br />Now for Visual C++ 2010 using vector:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>17.8</td></tr><tr><td>C++</td><td>18.7</td></tr><tr><td>Iterator</td><td>180.6</td></tr><tr><td>Assign</td><td>159.5</td></tr><tr><td>Copy</td><td>165.6</td></tr></table><br /><br />And string:<br /><table><tr><th>Method</th><th>Duration</th></tr><tr><td>C</td><td>16.5</td></tr><tr><td>C++</td><td>20.4</td></tr><tr><td>Iterator</td><td>224.4<td></td></tr><tr><td>Assign</td><td>222.8</td></tr><tr><td>Copy</td><td>320</td></tr></table><br /><br />We see here that the Copy method, which uses push_back() got a huge performance improvement. This seems to indicate that the STL implementation adds a 0 at the end of each operation, especially push_back(), instead of just when c_str() is called. Otherwise, string is faster.<br /><br />It's also sad to see that GCC while winning all the cases where iterators were involved, was significantly slower in all the straight forward cases. This seems to indicate that GCC has the smartest optimizations, but fails to optimize well when the logic is straightforward. Someone should look into that.<br /><br />It seems if you're trying to hold a collection of bytes, or whatever your wchar_t is, but don't care about the specialties of any particular container, as long as you don't push_back() a lot, string seems to be faster.<br /><br />Finally, here's a table of all the compilers and methods I tested ordered by speed:<br /><br /><table><br /><tr><th>Method</th><th>Duration</th></tr><tr><td>POSIX</td><td>5</td></tr><tr><td>LLVM 3.0 s C/C++</td><td>7.5</td></tr><tr><td>LLVM 3.0 v C/C++</td><td>8</td></tr><tr><td>MMAP</td><td>9</td></tr><tr><td>VC 2010 s C</td><td>16.5</td></tr><tr><td>VC 2010 v C</td><td>17.8</td></tr><tr><td>VC 2005 s C</td><td>18.3</td></tr><tr><td>VC 2010 v C++</td><td>19.7</td></tr><tr><td>VC 2010 s C++</td><td>20.4</td></tr><tr><td>VC 2005 s C++</td><td>21</td></tr><tr><td>GCC 4.6 v C++</td><td>22.8</td></tr><tr><td>GCC 4.6 v C</td><td>23.5</td></tr><tr><td>VC 2005 v C</td><td>24</td></tr><tr><td>GCC 4.6 s C/C++</td><td>24.5</td></tr><tr><td>VC 2005 v C++</td><td>26</td></tr><tr><td>LLVM 3.0 s Rdbuf</td><td>31.5</td></tr><tr><td>GCC 4.6 s Rdbuf</td><td>32.5</td></tr><tr><td>GCC 4.6 s Copy</td><td>63</td></tr><tr><td>GCC 4.6 s Iterator</td><td>64.5</td></tr><tr><td>GCC 4.6 s Assign</td><td>68</td></tr><tr><td>GCC 4.6 v Copy</td><td>68</td></tr><tr><td>GCC 4.6 v Iterator</td><td>73</td></tr><tr><td>GCC 4.6 v Assign</td><td>81.8</td></tr><tr><td>LLVM 3.0 s Copy</td><td>97</td></tr><tr><td>LLVM 3.0 s Assign</td><td>102</td></tr><tr><td>LLVM 3.0 s Iterator</td><td>110</td></tr><tr><td>VC 2010 v Assign</td><td>159.5</td></tr><tr><td>VC 2010 v Copy</td><td>165.6</td></tr><tr><td>VC 2005 v Copy</td><td>172</td></tr><tr><td>VC 2010 s Rdbuf</td><td>176.2</td></tr><tr><td>VC 2010 v Iterator</td><td>180.6</td></tr><tr><td>VC 2005 s Rdbuf</td><td>199</td></tr><tr><td>VC 2005 s Iterator</td><td>209.3</td></tr><tr><td>VC 2005 s Assign</td><td>221</td></tr><tr><td>VC 2010 s Assign</td><td>222.8</td></tr><tr><td>VC 2010 s Iterator</td><td>224.4</td></tr><tr><td>VC 2010 s Copy</td><td>320</td></tr><tr><td>VC 2005 v Iterator</td><td>370</td></tr><tr><td>VC 2005 v Assign</td><td>378</td></tr><tr><td>VC 2005 s Copy</td><td>483.5</td></tr><tr><td>LLVM 3.0 v Iterator</td><td>860</td></tr><tr><td>LLVM 3.0 v Copy</td><td>930</td></tr><tr><td>LLVM 3.0 v Assign</td><td>1328</td></tr></table>insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com15tag:blogger.com,1999:blog-833174317742362874.post-338111309388842472011-11-22T15:27:00.001-08:002013-04-02T19:02:30.474-07:00How to read in a file in C++So here's a simple question, what is the correct way to read in a file completely in C++?<br />
<br />
Various people have various solutions, those who use the C API, C++ API, or some variation of tricks with iterators and algorithms. Wondering which method is the fastest, I thought I might as well put the various options to the test, and the results were surprising.<br />
<br />
First, let me propose an API that we'll be using for the function. We'll send a function a C string (char *) of a filename, and we'll get back a C++ string (std::string) of the file contents. If the file cannot be opened, we'll throw an error why that is so. Of course you're welcome to change these functions to receive and return whatever format you prefer, but this is the prototype we'll be operating on:<br />
<pre>std::string get_file_contents(const char *filename);</pre>
<br />
Our first technique to consider is using the C API to read directly into a string. We'll open a file using fopen(), calculate the file size by seeking to the end, and then size a string appropriately. We'll read the contents into the string, and then return it.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <string></span>
<span style="color: green;">#include <cstdio></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::FILE *fp = std::fopen(filename, </span><span style="color: #dd0000;">"rb"</span><span style="color: black;">);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (fp)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> std::string contents;</span>
<span style="color: black;"> std::fseek(fp, </span><span style="color: blue;">0</span><span style="color: black;">, SEEK_END);</span>
<span style="color: black;"> contents.resize(std::ftell(fp));</span>
<span style="color: black;"> std::rewind(fp);</span>
<span style="color: black;"> std::fread(&contents[</span><span style="color: blue;">0</span><span style="color: black;">], </span><span style="color: blue;">1</span><span style="color: black;">, contents.size(), fp);</span>
<span style="color: black;"> std::fclose(fp);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(contents);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
I'm dubbing this technique "method C". This is more or less the technique of any proficient C++ programmer who prefers C style I/O would look like.<br />
<br />
The next technique we'll review is basically the same idea, but using C++ streams instead.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <fstream></span>
<span style="color: green;">#include <string></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::ifstream in(filename, std::ios::in | std::ios::binary);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (in)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> std::string contents;</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::end);</span>
<span style="color: black;"> contents.resize(in.tellg());</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::beg);</span>
<span style="color: black;"> in.read(&contents[</span><span style="color: blue;">0</span><span style="color: black;">], contents.size());</span>
<span style="color: black;"> in.close();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(contents);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
I'm dubbing this technique "method C++". Again, more or less a straight forward C++ implementation based on the same principals as before.<br />
<br />
The next technique people consider is using istreambuf_iterator. This iterator is designed for really fast iteration out of stream buffers (files) in C++.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <fstream></span>
<span style="color: green;">#include <streambuf></span>
<span style="color: green;">#include <string></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::ifstream in(filename, std::ios::in | std::ios::binary);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (in)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(std::string((std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>(in)), std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>()));</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
This method is liked by many because of how little code is needed to implement it, and you can read a file directly into all sorts of containers, not just strings. The method was also popularized by the <a href="http://www.amazon.com/gp/product/0201749629/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0201749629&linkCode=as2&tag=insacodi-20">Effective STL</a><img alt="" border="0" class="hioyqmdriwomkxxppfkw" height="1" src="http://www.assoc-amazon.com/e/ir?t=insacodi-20&l=as2&o=1&a=0201749629" style="border: none !important; margin: 0px !important;" width="1" /> book. I'm dubbing the technique "method iterator".<br />
<br />
Now some have looked at the last technique, and felt it could be optimized further, since if the string has an idea in advance how big it needs to be, it will reallocate less. So the idea is to reserve the size of the string, then pull the data in.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <fstream></span>
<span style="color: green;">#include <streambuf></span>
<span style="color: green;">#include <string></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::ifstream in(filename, std::ios::in | std::ios::binary);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (in)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> std::string contents;</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::end);</span>
<span style="color: black;"> contents.reserve(in.tellg());</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::beg);</span>
<span style="color: black;"> contents.assign((std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>(in)), std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>());</span>
<span style="color: black;"> in.close();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(contents);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
I will call this technique "method assign", since it uses the string's assign function.<br />
<br />
Some have questioned the previous function, as assign() in some implementations may very well replace the internal buffer, and therefore not benefit from reserving. Better to call push_back() instead, which will keep the existing buffer if no reallocation is needed.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <fstream></span>
<span style="color: green;">#include <streambuf></span>
<span style="color: green;">#include <string></span>
<span style="color: green;">#include <algorithm></span>
<span style="color: green;">#include <iterator></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::ifstream in(filename, std::ios::in | std::ios::binary);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (in)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> std::string contents;</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::end);</span>
<span style="color: black;"> contents.reserve(in.tellg());</span>
<span style="color: black;"> in.seekg(</span><span style="color: blue;">0</span><span style="color: black;">, std::ios::beg);</span>
<span style="color: black;"> std::copy((std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>(in)), std::istreambuf_iterator<</span><span style="color: maroon;">char</span><span style="color: black;">>(), std::back_inserter(contents));</span>
<span style="color: black;"> in.close();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(contents);</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
Combining std::copy() and std::back_inserter(), we can achieve our goal. I'm labeling this technique "method copy".<br />
<br />
Lastly, some want to try another approach entirely. C++ streams have some very fast copying to another stream via operator<< on their internal buffers. Therefore, we can copy directly into a string stream, and then return the string that string stream uses.<br />
<pre style="background-color: #f5f2d5;"><span style="color: green;">#include <fstream></span>
<span style="color: green;">#include <sstream></span>
<span style="color: green;">#include <string></span>
<span style="color: green;">#include <cerrno></span>
<span style="color: black;">std::string get_file_contents(</span><span style="color: maroon;">const</span><span style="color: black;"> </span><span style="color: maroon;">char</span><span style="color: black;"> *filename)</span>
<span style="color: black;">{</span>
<span style="color: black;"> std::ifstream in(filename, std::ios::in | std::ios::binary);</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">if</span><span style="color: black;"> (in)</span>
<span style="color: black;"> {</span>
<span style="color: black;"> std::ostringstream contents;</span>
<span style="color: black;"> contents << in.rdbuf();</span>
<span style="color: black;"> in.close();</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">return</span><span style="color: black;">(contents.str());</span>
<span style="color: black;"> }</span>
<span style="color: black;"> </span><span style="color: black; font-weight: bold;">throw</span><span style="color: black;">(errno);</span>
<span style="color: black;">}</span></pre>
<br />
We'll call this technique "method rdbuf".<br />
<br />
<br />
Now which is the fastest method to use if all you actually want to do is read the file into a string and return it? The exact speeds in relation to each other may vary from one implementation to another, but the overall margins between the various techniques should be similar.<br />
<br />
I conducted my tests with libstdc++ and GCC 4.6, what you see may vary from this.<br />
<br />
I tested with multiple megabyte files, reading in one after another, and repeated the tests a dozen times and averaged the results.<br />
<br />
<br />
<table><tbody>
<tr><th>Method</th><th>Duration</th></tr>
<tr><td>C</td><td>24.5</td></tr>
<tr><td>C++</td><td>24.5</td></tr>
<tr><td>Iterator</td><td>64.5</td></tr>
<tr><td>Assign</td><td>68</td></tr>
<tr><td>Copy</td><td>62.5</td></tr>
<tr><td>Rdbuf</td><td>32.5</td></tr>
</tbody></table>
<br />
<br />
Ordered by speed:<br />
<br />
<br />
<table><tbody>
<tr><th>Method</th><th>Duration</th></tr>
<tr><td>C/C++</td><td>24.5</td></tr>
<tr><td>Rdbuf</td><td>32.5</td></tr>
<tr><td>Copy</td><td>62.5</td></tr>
<tr><td>Iterator</td><td>64.5</td></tr>
<tr><td>Assign</td><td>68</td></tr>
</tbody></table>
<br />
<br />
These results are rather interesting. There was no speed difference at all whether using the C or C++ API for reading a file. This should be obvious to us all, but yet many people strangely think that the C API has less overhead. The straight forward vanilla methods were also faster than anything involving iterators.<br />
<br />
C++ stream to stream copying is really fast. It probably only took a bit longer than the vanilla method due to some reallocations needed. If you're doing disk file to disk file though, you probably want to consider this option, and go directly from in stream to out stream.<br />
<br />
Using the istreambuf_iterator methods while popular and concise are actually rather slow. Sure they're faster than istream_iterators (with skipping turned off), but they can't compete with more direct methods.<br />
<br />
A C++ string's internal assign() function, at least in libstdc++, seems to throw away the existing buffer (at the time of this writing), so reserving then assigning is rather useless. On the other hand, reading directly into a string, or a different container for that matter, isn't necessarily your most optimal solution where iterators are concerned. Using the external std::copy() function, along with back inserting after reservation is faster than straight up initialization. You might want to consider this method for inserting into some other containers. In fact, I found that std::copy() of istreambuf_iterators with back inserter into an std::deque to be faster than straight up initialization (81 vs 88.5), despite a Deque not being able to reserve room in advance (nor does such make sense with a Deque).<br />
<br />
I also found this to be a cute way to get a file into a container backwards, despite a Deque being rather useless for working with file contents.<br />
<pre>std::deque<char> contents;
std::copy((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>(), std::front_inserter(contents));</pre>
<br />
Now go out there and speed up your applications!<br />
<br />
If there's any demand, I'll see about performing these tests with other C++ implementations.insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com82tag:blogger.com,1999:blog-833174317742362874.post-4741534685290863242011-11-19T12:18:00.001-08:002011-11-19T12:42:17.228-08:00Making modconf work with Linux 3<p>If you want a nice curses based GUI to add and remove modules from Linux on the fly, you probably used modconf. They say it's deprecated, but they still haven't made anything to replace it.</p><p>I noticed that it stopped working on Linux 3.</p><p>The problem is as follows, the configuration scripts that come with modconf generally look to see if you're using Linux 2.0 - 2.4 and do one thing, and do another for 2.5 and up. They generally check just the second digit, so now with Linux "3.0" it thinks you're using 2.0 and does something for <2.5 when it should be using the >=2.5 method.</p><p>Edit <i>/usr/share/modconf/params</i><br />Look for:<br /><pre>case "$(uname -r | cut -f2 -d.)" in<br /> 0|1|2|3|4)<br /> CFGFILE=$Target/etc/modules.conf<br /> MODUTILSDIR=$Target/etc/modutils<br /> ;;<br /> *)<br /> CFGFILE=$Target/etc/modprobe.d<br /> MODUTILSDIR=$Target/etc/modprobe.d<br /> ;;<br />esac<br /></pre><br />Replace it with:<br /><pre>CFGFILE=$Target/etc/modprobe.d<br />MODUTILSDIR=$Target/etc/modprobe.d<br /></pre></p><p>Edit <i>/usr/share/modconf/util</i><br />Look for:<br /><pre> case "$(uname -r | cut -f2 -d.)" in<br /> 0|1|2|3|4)<br /> modsuffix=".o"<br /> using_mit=1<br /> ;;<br /> *) modsuffix=".ko" ;;<br /> esac<br /></pre><br />Replace it with:<br /><pre>modsuffix=".ko"<br /></pre></p><p>And that's all there is to it!</p>insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com4tag:blogger.com,1999:blog-833174317742362874.post-43608716714308036122011-10-25T18:24:00.000-07:002013-03-18T13:27:20.249-07:00A stronger C/C++ PreprocessorEver felt you needed some preprocessing to generate some C/C++ code for you, but the C preprocesssor is too lacking? Say for example you wanted to include <b>part</b> of another file, or include output from an application. Perhaps the compile should be based on some data found on a remote server? Well, there are stronger preprocessors which work for C/C++, such as PHP.<br />
<br />
Here's an example:<br />
<pre>/tmp> php test.cpp.php | g++ -x c++ -Wall -o test -
/tmp> ./test
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
/tmp> cat test.cpp.php
#include <iostream>
int main()
{
<?php for ($i = 0; $i < 5; ++$i) { echo 'std::cout << "Hello ', $i, '" << std::endl;'; } ?>
return(0);
}
/tmp></pre>
Here's a more interesting example:<br />
<pre>/tmp> php weather_example.c.php | gcc -x c -Wall -o weather_example -
/tmp> ./weather_example
Hi, when I was compiled, the weather here in New York City was 57F
/tmp> cat weather_example.c.php
<?php
$w = file_get_contents('http://www.google.com/ig/api?weather=New+York+City');
$f = '<temp_f data="';
echo '#define __WEATHER__ ', (int)substr($w, strpos($w, $f)+strlen($f)), 'U', "\n";
?>
#include <stdio.h>
int main()
{
printf("Hi, when I was compiled, the weather here in New York City was %uF\n", __WEATHER__);
return(0);
}
/tmp></pre>
insane coderhttp://www.blogger.com/profile/06901386115570670209noreply@blogger.com7