Effective Modern C++ Quick Review and Sale

For those of you familiar with Scott Meyers amazing Effective C++ and Effective STL (both of which I highly recommend to C++ programmers), Effective Modern C++ is now out which covers best practices for new features introduced in C++ 2011, as well as some upcoming features of C++ 2014.

For one more day only, you can also get $10 off any of the above links to Amazon if you use the code BOOKDEAL25 when you checkout (if your order totals $40 or more).

For those of you unfamiliar with the series, I'll provide you with a quick review.

Effective C++ is an excellent book for C++ beginners and experts alike. It helps beginners understand what is and how to choose the best tool for the job when C++ offers multiple ways of doing things. It can also help even experts to better understand how the language works, and what is and isn't good practice. You'll learn how to correctly navigate this large language, and end up writing programs where memory management or other resource issues are a thing of the past. You'll learn how to design programming interfaces that look sane, and you won't come back to them later asking yourself what drugs you were on when you originally wrote the code.

Effective STL dives more into the standard library of C++, helping the developer understand which container or algorithm is the best tool for the job, and how to use them most effectively. If you've ever wondered to yourself if you should use an std::vector or std::list, you need to read this book. If you've ever wondered what functors are or why bother with features which at first glance may seem like a roundabout way of doing things, you'll unlock the true potential of these features and more, and learn how what may seem weird to the untrained eye is actually powerful and produces better code.

Effective Modern C++ introduces the reader to new features added to the language in recent years, and as the other books in the series, helps them to better understand them, get the best out of them, and use them correctly. You'll learn how to choose and make the best out of the new resource management introduced with std::shared_ptr, std::unique_ptr, and std::weak_ptr, avoid common pitfalls with the new threading library, new functional programming techniques exposed in modern C++, as well as how to achieve even more optimization with all the new features.

The series also includes More Effective C++, however I do not recommend this book. Unlike the previously mentioned books, the latest edition of More Effective C++ came out before C++ was even standardized, and has been out of date for well over a decade. It does explain certain features of the language, most of which are still present today, but those features are best covered by an introduction to the language itself like The C++ Programming Language (hardcover), Principals and Practice Using C++, and The C++ Standard Library. One thing More Effective C++ covers that isn't in these great C++ learning and reference books is how to use C and C++ together effectively. Although in my opinion that's better covered in the more C focused book Expert C Programming: Deep C Secrets.

For those of you thinking to yourself that C++ is a horrible language, especially since it has so many books printed for it on how to correctly use it and get the best out of it, you might want to realize that most well established languages have such books. For example, Effective Java, Effective JavaScript, and the groundbreaking JavaScript: The Good Parts. Proper programming design is an art, and one could even go as far to say that if a book of this nature does not exist for a language, it's because no one has yet figured out how to really use it effectively.

Memory management in C and auto allocating sprintf() - asprintf()

Memory Management

Memory management in C is viewed by some to be quite tricky. One needs to work with pointers that can point anywhere in memory, and if misused, cause a program to misbehave, or worse.

The basic functions to allocate and deallocate memory in C are malloc() and free() respectively. The malloc() function takes a size in bytes of how much to allocate, and returns a pointer to the allocated memory to be used. Upon failure, a null pointer, which can be thought of as a pointer pointing to 0 is returned. The free() function takes the pointer returned by malloc(), and deallocates the memory, or frees up the memory it was once pointing to.

To work with malloc() in simple situations, typically, code along the following lines is used:

void *p = malloc(size);
if (p)
{
  ... work with p ...
  free(p);
}
else
{
  ... handle error scenario ...
}

Unfortunately many experienced programmers forget to handle the failure scenario. I've even heard some say they purposely don't, as they have no clue how to proceed, and just letting the program crash is good enough for them. If you meet someone who makes that argument, revoke their programming license. We don't need such near sighted idiots writing libraries.

In any case, even the above can lead to some misuse. After this block of code runs, what is p now pointing to?

After the above code runs, in the case that malloc() succeeded, p is now pointing to memory in middle of nowhere, and can't be used. This is known as a dangling pointer. Dangling pointers can be dangerous, as an if clause as above will think the pointer is valid, and may do something with it, or lead to the infamous use after free bug. This becomes more likely to occur as the situation becomes more complicated and there are loops involved, and how malloc() and free() interact can take multiple paths.

Pointers involved with memory management should always be pointing at 0 or at allocated memory. Anything else is just asking for trouble. Therefore, I deem any direct use of free() dangerous, as it doesn't set the pointer to 0.

So if free() is considered harmful, what should one use?

In C++, I recommend the following:

static inline void insane_free(void *&p)
{
  free(p);
  p = 0;
}

This insane_free() is now a drop in replacement for free(), and can be used instead. (Since C++ programs normally use new and delete/delete[] instead, I leave it as an exercise to the reader how to work with those.)

However, C doesn't support direct references. One can pass a pointer by a pointer to accomplish similar results, but that becomes clunky and is not a drop in replacement. So in C, I recommend the following:

#define insane_free(p) { free(p); p = 0; }

It makes use of the preprocessor, so some may consider it messy, but it can be used wherever free() currently is. One could also name the macro free in order to automatically replace existing code, but it's best not to program that way, as you begin to rely on these semantics. This in turn means someone copying your code may think a call to free() is the normal free() and not realize something special is occurring when they copy it elsewhere without the macro.

Correct usage in simple cases is then:

void *p = malloc(size);
if (p)
{
  ... work with p ...
  insane_free(p);
}
else
{
  ... handle error scenario ...
}

If you think using a wrapper macro or function is overkill, and just always manually assigning the pointer to 0 after freeing is the way to go, consider that it's unwieldy to constantly do so, and you may forget to. If the above technique was always used, all use after free bugs would never have occurred in the first place.

Something else to be aware of is that there's nothing wrong with calling free(0). However, calling free() upon a pointer which is not null and not pointing to allocated memory is forbidden and will crash your program. So stick to the advice here, and you may just find memory management became significantly easier.

If all this talk of pointers is beyond you, consider acquiring Understanding and Using C Pointers.

sprintf() and asprintf()

If you do a lot of C programming, at some point, you probably have used the sprintf() function, or its safer counterpart snprintf().

These two functions sprintf() and snprintf() act like printf(), but instead of printing to the standard output, they print to a fixed-length string buffer. Now a fixed-length string buffer is great and all, but what if you wanted something which automatically allocated the amount it needed?

Enter asprintf(), a function which acts like sprintf(), but is auto-allocating, and needs no buffer supplied. This function was invented ages ago by GLIBC, shortly thereafter copied to the modern BSDs, and found its way further still into all sorts of libraries, although is not yet ubiquitous.

Let's compare the prototypes of the two:

int sprintf(char *buffer, const char *format, ...); 

int asprintf(char **ret, const char *format, ...);

The sanest approach would have been for a function like asprintf() to have the prototype of:

char *asprintf(const char *format, ...);

But its creators wanted to make it act like sprintf(), and its design can also be potentially more useful.

Instead of passing asprintf() a buffer, a pointer to a variable of type char * needs to be passed, like so:

char *buffer;
asprintf(&buffer, ...whatever...);

Now how asprintf() actually works is no big secret. The C99 standard specified that snprintf() upon failure should return the amount of characters that would be needed to contain its output. Which means that conceptually something along the following lines would be all that asprintf() needs to do:

char *buffer = malloc(snprintf(0, 0, format, data...)+1);
sprintf(buffer, format, data...);

Of course though, the above taken verbatim would be incorrect, because it mistakenly assumes that nothing can go wrong, such as the malloc() or snprintf() failing.

First let's better understand what the *printf() functions return. Upon success, they return the amount of characters written to the string (which does not include the trailing null byte). Or in other words, the return value is equivalent to calling strlen() on the data being output, which can save you needing to use a strlen() call with sprintf() or similar functions for certain scenarios. Upon failure, for whatever reason, the return is -1. Of course there's the above mentioned exception to this with snprintf(), where the amount of characters needed to contain the output would be returned instead. If during the output, the size overflows (exceeds INT_MAX), many implementations will return a large negative value (failure with snprintf(), or success with all the functions).

Like the other functions, asprintf() also returns an integer of the nature described above. Which means working with asprintf() should go something like this:

char *buffer;
if (asprintf(&buffer, ...whatever...) != -1)
{
  do_whatever(buffer);
  insane_free(buffer);
}

However, unlike the other functions, asprintf() has a second return value, its first argument, or what the function sees as *ret. To comply with the memory management discussion above, this should also be set to 0 upon failure. Unfortunately, many popular implementations, including those in GLIBC and MinGW fail to do so.

Since I develop with the above systems, and I'm using asprintf() in loops with multiple paths, it becomes unwieldy to need to pass around the buffer and the returned int, so I'd of course want saner semantics which don't leave dangling pointers in my program.

In order to correct such mistakes, I would need to take code from elsewhere, or whip up my own function. Now I find developing functions such as these to be relatively simple, but even so, I always go to check other implementations to see if there's any important points I'm missing before I go implement one. Maybe, I'll even find one which meets my standards with a decent license which I can just copy verbatim.

In researching this, to my shock and horror, I came across implementations which properly ensure that *ret is set to 0 upon failure, but the returned int may be undefined in certain cases. That some of the most popular implementations get one half wrong, and that some of the less popular get the other half wrong is just downright terrifying. This means that there isn't any necessarily portable way to check for failure with the different implementations. I certainly was not expecting that, but with the amount of horrible code out there, I guess I really shouldn't be surprised anymore.

Also in the course of research, besides finding many implementations taking a non-portable approach, many have problems in all sorts of edge cases. Such as mishandling overflow, or not realizing that two successive calls to a *printf() function with the same data may not necessarily yield the same results. Some try to calculate the length needed with some extra logic and only call sprintf() once, but this logic may not be portable, or always needs updating as new types are added to the format string as standards progress, or the C library decided to offer new features. Some of the mistakes I found seem to be due to expecting a certain implementation of underlying functions, and then later the underlying functions were altered, or the code was copied verbatim to another library, without noting the underlying functions acted differently.

So, once again, I'm finding myself needing to supply the world with more usable implementations.

Let's dive into how to implement asprinf().

Every one of these kind of functions actually has two variants, the regular which takes an unlimited amount of arguments, and the v variants which take a va_list (defined in stdarg.h) after the format argument instead. These va_lists are what ... gets turned into after use, and in fact, every non-v *printf() function is actually wrapped to a counterpart v*printf() function. This makes implementing asprintf() itself quite straight forward:

To fix the aforementioned return problems, one could also easily throw in here a check upon the correct return variable used in the underlying vasprintf() implementation and use it to set the other. However, that's not a very portable fix, and the underlying implementation of vasprintf() can have other issues as described above.

A straight forward implementation of vasprintf() would be:

As long as you have a proper C99 implementation of stdarg.h and vsnprintf(), you should be good to go. However, some systems may have vsnprintf() but not va_copy(). The va_copy() macro is needed because a va_list may not just be a simple object, but have handles to elsewhere, and a deep copy is needed. Since vsnprintf() being passed the original va_list may modify its contents, a copy is needed because the function is called twice.

Microsoft Visual C++ (MSVC, or Microsoft Vs. C++ as I like to think of it) up until the latest versions has utterly lacked va_copy(). This and several other  systems that lack it though usually have simple va_lists that can be shallow copied. To gain compatibility with them, simply employ:

#ifndef va_copy 

#define va_copy(dest, src) dest = src 

#endif

Be warned though that if your system lacks va_copy(), and a deep copy is required, using the above is a recipe for disaster.

Once we're dealing with systems where shallow copy works though, the following works just as well, as vsnprintf() will be copying the va_list it receives and won't be modifying other data.

Before we go further, there's two points I'd like to make.

• Some implementations of vsnprintf() are wrong, and always return -1 upon failure, not the size that would've been needed. On such systems, another approach will need to be taken to calculate the length required, and the implementations here of vasprintf() (and by extension asprintf()) will just always return -1 and *ret (or *strp) will be 0.
• The code if ((r < 0) || (r > size)) could instead be if (r != size), more on that later.

Now on Windows, vsnprintf() always returns -1 upon failure, in violation of the C99 standard. However, in violation of Microsoft's own specifications, and undocumented, I found that vsnprintf() with the first two parameters being passed 0 as in the above code actually works correctly. It's only when you're passing data there that the Windows implementation violates the spec. But in any case, relying on undocumented behavior is never a good idea.

On certain versions of MinGW, if __USE_MINGW_ANSI_STDIO is defined before stdio.h is included, it'll cause the broken Windows *printf() functions to be replaced with C99 standards compliant ones.

In any case though, Windows actually provides a different function to retrieve the needed length, _vscprintf(). A simple implementation using it would be:

This however makes the mistake of assuming that vsnprintf() is implemented incorrectly as it currently is with MSVC. Meaning this will break if Microsoft ever fixes the function, or you're using MinGW with __USE_MINGW_ANSI_STDIO. So better to use:

Lastly, let me return to that second point from earlier. The vsnprintf() function call the second time may fail because the system ran out of memory to perform its activities once the call to malloc() succeeds, or something else happens on the system to cause it to fail. But also, in a multi-threaded program, the various arguments being passed could have their data change between the two calls.

Now if you're calling functions while another thread is modifying the same variables you're passing to said function, you're just asking for trouble. Personally, I think that all the final check should do is ensure that r is equal to size, and if not, something went wrong, free the data (with insane_free() of course), and set r to -1. However, any value between 0 and size (inclusive), even when not equal to size means the call succeeded for some definition of success, which the above implementations all allow for (except where not possible Microsoft). Based on this idea, several of the implementations I looked at constantly loop while vsnprintf() continues to indicate that the buffer needs to be larger. Therefore, I'll provide such an implementation as well:

Like the first implementation, if all you lacked was va_copy(), and shallow copy is fine, it's easy to get this to work on your platform as described above. But if vsnprintf() isn't implemented correctly (hello MSVC), this will always fail.

All the code here including the appropriate headers, along with notes and usage examples are all packed up and ready to use on my asprintf() implementation website. Between everything offered, you should hopefully find something that works well for you, and is better than what your platform provides, or alternative junk out there.

As always, I'm only human, so if you found any bugs, please inform me.

Avoid incorrect ChaCha20 implementations

ChaCha20 is a stream cipher which is gaining a lot of popularity of late. Practically every library today which provides ciphers seems to have it as an addition in their latest releases.

In cryptography, there are two kinds of ciphers, block ciphers and stream ciphers. Block ciphers are where the underlying algorithm works with data with a certain fixed chunk size (or block). Popular blocks sizes are 16 and 64 bytes. Stream ciphers are effectively block ciphers where the chunk size is a single byte.

Classical stream ciphers, such as RC4, can work with data of arbitrary size, although every single byte is dependent on every previous byte. Which means encryption/decryption cannot begin in the middle of some data, and maintain compatibility where some other starting point was used. Block ciphers generally can have their blocks encrypted and decrypted arbitrarily, with none dependent upon any other, however, they cannot work with data of arbitrary size.

In order to allow block ciphers to work with data of arbitrary size, one needs to pad the data to be encrypted to a multiple of the block size. However, a clever alternative is counter mode.

Different modes for working with block ciphers exist. Some try to improve security by making each block depend on every other, some utilize various interesting techniques for other properties.  Counter mode does not encrypt the desired data (the plaintext) directly, rather, an ever incrementing counter is encrypted. The result of this encryption is then xored with the desired data.

Counter mode effectively turns a block cipher into a stream cipher, as the plaintext is never actually passed to the block cipher. Rather, a counter which is a multiple of the block size is used. One can always xor bytes with an arbitrary size, and since that is the only step in counter mode against the plain text, it is effectively a stream cipher. Since the underlying cipher can be a block cipher with no dependency between blocks, this kind of stream cipher also allows one to jump ahead to any particular multiple of the block size in the data, and begin encryption/decryption from there.

Now while ChaCha20 is advertised as a stream cipher, it's actually designed as a block cipher in counter mode. The internal design mostly mirrors that of typical counter mode design, except that the counter components are directly fused with a large but simple block cipher. Since it's really a block cipher, it has an internal block size, and also allows one to jump ahead to some multiple of it.

Since ChaCha20 is considered to have a great level of security, and all these other wonderful properties, it's starting to see a lot of use. However, practically every implementation I'm seeing is either utterly broken, or has some ridiculous API.

Common ChaCha20 implementation mistakes:

• Implemented as a typical block cipher, not allowing usage with arbitrary amounts of bytes, or worse, the API allows for it, but produces incorrect results.
• Implemented as a typical stream cipher with no way to jump ahead.
• Failing on Big-Endian systems.

The first mistake I listed is the most common. If some software is only using ChaCha20 internally, and always using it in a multiple of its block size (or it's all the crummy API offers), then things are fine. But if it's a library which is inviting others to use it, and it can be used incorrectly, expect disaster to ensue.

The reference implementation of ChaCha20 was designed that an arbitrary amount of data can be encrypted, as long as all but the last usage of the API was a multiple of the block size. This was also mentioned in its documentation. However, practically every other implementation out there copies this design in some way, but makes no note of it. Worse yet, some libraries are offering ChaCha20 with this implementation flaw alongside other stream ciphers with an identical API whereas those can be used arbitrarily throughout.

Essentially, this means if you're using ChaCha20 right now in a continuous fashion with chunks of various sizes, your data is being encrypted incorrectly, and won't be interoperable with other implementations. These broken implementations are able to output exactly one chunk correctly which is not a multiple of the block size, which destroys their internal buffers, and screws up every output thereafter.

I noticed a similar situation with hash algorithm implementations several years back. However, most hash implementations are fine. Yet with ChaCha20, practically every implementation I looked at recently was broken.

Since this situation cannot stand, especially with ChaCha20 gaining speed, I am providing a simple implementation without these flaws. This implementation is designed to be correct, portable, and simple. (Those wanting an optimized version of this should consider paying for more optimized routines)

Usage of the C99 API I designed is as follows:

Custom type: chacha20_ctx
This type is used as a context for a state of encryption.

To initialize:

void chacha20_setup(chacha20_ctx *ctx, const uint8_t *key, size_t length, uint8_t nonce[8]);

The encryption key is passed via a pointer to a byte array and its length in bytes. The key can be 16 or 32 bytes. The nonce is always 8 bytes.

Once initialized, to encrypt data:

void chacha20_encrypt(chacha20_ctx *ctx, const uint8_t *in, uint8_t *out, size_t length);

You can pass an arbitrary amount of data to be encrypted, just ensure the output buffer is always at least as large as the input buffer. This function can be called repeatedly, and it doesn't matter what was done with it previously.

To decrypt data, initialize, and then call the decryption function:

void chacha20_decrypt(chacha20_ctx *ctx, const uint8_t *in, uint8_t *out, size_t length);

For encryption or decryption, if you want to jump ahead to a particular block:

void chacha20_counter_set(chacha20_ctx *ctx, uint64_t counter);

Counter is essentially the number of the next block to encrypt/decrypt. ChaCha20's internal block size is 64 bytes, so to calculate how many bytes are skipped by a particular counter value, multiply it by 64.

In addition to just providing a library, I gathered the test vectors that were submitted for various RFCs, and included a series of unit tests to test it for correctness.

For fun, since I'm also playing around a bit with LibreSSL these days, I wrapped its API up in the API I described above. The wrapper is included in my package with the rest of the code, however it is currently not designed for serious usage outside of the included test cases.

Since I already whipped up some unit tests that anyone can use, I'll leave it as an exercise to the reader to determine which libraries are and aren't implemented correctly.

I tried to ensure my library is bug free, but I am only human. If you find a mistake, please report it.

Protecting private keys

Web servers use private keys which they alone have in order to secure connections with users. Private keys must be protected at all costs.

In order to protect private keys on disk, one generally encrypts them with a password, which is then needed by the web server upon launch in order to decrypt and use it in memory. However, if measures aren't taken to secure the memory containing the private key, it can be stolen from there too, which would be catastrophic.

Normally, one doesn't need to worry about outsiders getting a hold of data from memory unless the attackers have direct access to the server itself. But bugs like Heartbleed allow remote users to grab random data from memory. Once data is in memory, the application and all the libraries it uses could divulge secret data if there is a buffer overflow lurking somewhere.

To protect against exploiting such bugs, one should ensure that buffer overflows do not have access to memory containing private data. The memory containing private keys and similar kinds of data should be protected, meaning nothing should be allowed to read from them, not even the web server itself.

Now obviously a program needs some access to a private key in order to work with it, so it can't just prevent all access from it. Rather, once a private key or similar data is loaded into memory, that memory should have its read permissions removed. When, and only when some activity needs to be performed with the private key, read permissions can be restored, the activity performed, and then read permissions revoked. This will ensure the rest of the application cannot access what it does not need, nor should be allowed to access.

On UNIX systems, one can use mprotect() to change the permission on a page of memory. On Windows, one can use VirtualProtect().

The above however has a crucial flaw - multi-threading. In a threaded application, all threads have access to data of other threads. So while one thread may be performing some critical private key related code, and allows read access for the moment, another thread can read it outside of the critical portion of code too. Therefore, even more isolation is needed.

To truly isolate the code that uses a private key and similar data, all the code that handles that stuff should be placed into its own process. The rest of the application can then request that well defined activities be performed via a pipe or other form of inter-process communication. This will also ensure that other kinds of bugs in the application, such as buffer overflows that allow arbitrary code execution cannot reestablish read access to the secret data.

On UNIX systems, one can use fork() to create a process which is still part of the same application. On all systems, the separate process can be a separate application with a well defined restrictive IPC API with limited and secured access by the web server.

No insecure library or silly bug in your web server should ever allow the application to divulge such secrets. If such services are not utilizing the techniques above, then they're just biding their time until the next Heartbleed.

Copying code != Copying implementation

A little over a week ago, I wrote an article about some common porting mistakes for a new library - LibreSSL. I've since received multiple questions about copying code from other projects. Mainly, if the original project uses certain code or another popular library is using some compatibility code, what is wrong with copying one of those directly?

The most obvious problem with copying from some compatibility library is that they may not be written properly. The primary concern behind most compatibility libraries is that things should compile. Just because things compile doesn't mean errors and all scenarios are being handled properly, or that the implementation is in any way secure. The popular libbsd and many GNU library shims may seem to get the job done, but only superficially.

However, the real problem is that copying some code does not mean you copied the implementation. There's a lot more to an implementation than just some code. For example, perhaps the source in question is being compiled with certain parameters or with special compilers, and the code is written specifically for that situation. If you copied the code but not the build environment, you only nabbed part of the implementation.

Another point to consider is if the code you copied is even used how you think it is. Let's look at a function from NetBSD:

int
consttime_memequal(const void *b1, const void *b2, size_t len)
{
 const char *c1 = b1, *c2 = b2;
 int res = 0;

 while (len --)
  res |= *c1++ ^ *c2++;

 /*
  * If the compiler for your favourite architecture generates a
  * conditional branch for `!res', it will be a data-dependent
  * branch, in which case this should be replaced by
  *
  * return (1 - (1 & ((res - 1) >> 8)));
  *
  * or rewritten in assembly.
  */
 return !res;
}

Note how the comment here says that this code may not function correctly on all architectures. In fact, some architectures may have an implementation in assembly which works correctly, whereas the C version does not.

NetBSD, like most C libraries, offers assembly variants for certain functions on certain architectures. If you copy a C file from it without the assembly files for the architecture in question, then the implementation was not copied, and in fact, may even produce incorrect results. You can't always depend on scanning comments either, as previous versions of this C file didn't contain the comment.

Now, let's say you copied files and their build environment, including possible assembly variants, does that now mean the implementation was definitively copied? No! Some functions used in the copied code may actually depend on certain characteristics within this particular implementation. Let me provide an example. Let's say consttime_memequal() was implemented as follows:

int consttime_memequal(const void *s1, const void *s2, size_t n)
{
   return !memcmp(s1, s2, n);
}

consttime_memequal() must run in a constant amount of time regardless if whether its two parameters are equal or not. Yet this version here wraps directly to memcmp(), which does not make such guarantees. Assuming this version was correct on the platform in question, it is correct because of an external factor, not because of the C code presented here.

Some compilers may provide their own version of memcmp() which they'll use instead of the C library's, such as GCC. Perhaps some compiler provides memcmp(), along with a compile option to make all memcmp() operations constant-time. The file containing this function can then be compiled with constant-time-memcmp enabled, and now everything else can call consttime_memequal() without any special compilers or build options.

Now while the above is possible, it's not the only possibility. Perhaps memcmp() on the platform in question just happens to be constant-time for some reason. In such a scenario, even if you used the exact same compiler with the exact same build options, and the exact same code, your implementation is still incorrect, because your underlying functions behave differently.

Bottom line, just copying code is incorrect. If some function is supposed to be doing something with certain properties beyond just fulfilling some activity (perhaps constant time, or defying compiler optimizations), then you must review the entire implementation and determine what aspect of it gives it this property. Without doing so, you're only fooling yourself into believing you copied an implementation.

Implementation copying checklist:

• Understand what you're attempting to copy, and what properties it carries.
• Ensure what you're copying actually performs its objectives.
• Ensure you copy the entire implementation.
• Ensure there's nothing about the compiler, compile options, or other aspects of the build environment which you forgot to copy.
• Ensure you copy any needed alternatives for certain architectures.
• Ensure the code does not depend on different implementations of functions you already have.
• Test everything for correctness, including matching output for error situations, and extra properties.

LibreSSL: The good and the bad

OpenSSL & LibreSSL

OpenBSD recently forked the popular SSL/TLS library OpenSSL into LibreSSL. Most of the reaction to this that I've seen tends to be pretty angry. People don't like the idea of a project being forked, they'd rather people work together, and have the OpenBSD team instead join OpenSSL.

Now, for those of you that don't know it, OpenSSL is at the same time the best and most popular SSL/TLS library available, and utter junk. It's the best because it covers a wide array of the capabilities that exist across the many standards that make up SSL/TLS, and has seen years of development to iron out a multitude of issues and attacks levied against the specifications. The developers also seem to know a lot more about programming than the developers behind some of the competing SSL/TLS libraries. There are a ton of gotchas when it comes to developing an SSL/TLS stack, way beyond other areas of development, and things must be done absolutely correctly.

Cryptographic development challenges

Aside from requiring meticulous programming to avoid typical programming issues and specific problems of the language in question, for SSL/TLS, one also needs to worry about:

• Ensuring libraries work correctly for tons of cases which are difficult to test for, and where unit tests cannot be exhaustive, and regression testing is far from complete.
• Work in a completely hostile environment where every single outside variable must be validated by itself, and as part of a larger whole.
• Data has to be correct both as raw data, and in their particular meaning.
• Ensure data dependent operations run in actual constant time despite differences in their values to avoid timing attacks.
• Ensure errors caught aren't actually handled immediately, but only at the very end of the single cryptographic unit as a whole completes, in order to avoid timing attacks. The rest of the unit must function normally despite the error(s).
• Random data must be taken from an unpredictable source, and random streams must not have any detectable patterns or biases to them.
• Sensitive/secret data must be protected, without allowing mathematical trickery or the computer environment somehow revealing them. 
• Ensuring data is wiped clean, without the compiler optimizations or virtual machine ignoring what they deem to be pointless operations.
• The inability to use some high-level languages because they lack a way to tie in forceful cleanup of primitive data types, and their error handling mechanisms may end up leaving no way to wipe data, or data is duplicated without permission.
• Almost every single thing which may be the right way of doing things elsewhere is completely wrong where cryptography is concerned.
• Things still have to be extremely well optimized, otherwise the expenses are too high to be viable in most scenarios.

The above list is hardly exhaustive, and is pretty generic. Various cryptographic algorithms also have a ton of specific issues with them that need to be avoided. These issues are not necessarily specified in the standards, but have been learned over time from those that made mistakes, or by various research.

Some examples of things to avoid which is part of the collective knowledge of SSL/TLS developers:

• Particular constants with certain algorithms must be avoided.
• Particular constants with certain algorithms must be used.
• Certain algorithms don't work well together.
• Strings must also be handled as raw data.
• Certain magnitudes are dangerous.
• Some platforms process certain algorithms or data in a way which must be worked around.
• Data broken up in certain ways is dangerous.

Now, one can read the specifications and implement them using best software engineering practices, and best cryptographic engineering practices, but where does one learn all the things to avoid? I've read 3 different books which cover how to implement significant parts or all of SSL/TLS, and each one listed unique gotchas that weren't in the other two. They're not mentioned in the various standards, and many of them are far from obvious. Unless you've spent years developing SSL/TLS, and took note of every mistake ever made, you're probably doing it wrong. In fact, most alternative SSL/TLS implementations I've seen make these various mistakes that OpenSSL already learned from.

Who should design cryptographic libraries

In order to create a proper SSL/TLS implementation you need to be a master of:

• Cryptographic algorithms.
• Cryptographic practice.
• Software engineering.
• Software optimization.
• The language(s) used.
• Domain specific knowledge.

Rarely are developers a true master at one of these, let alone being a master of the algorithms and software engineering. Which means most SSL/TLS libraries will either fail at being at the forefront of cryptography or fail nice sane design, or both.

This is also why OpenSSL is utter junk. Some of its developers may have been decent at the algorithms and software engineering, but not particulary good at either of them. Other developers may have been a master of one, but absolutely abysmal at the other. Due to its popularity, OpenSSL is also a dumping ground for every new cryptographic idea, good or bad, and is constantly pushed in every direction, being spread far too thin.

The OpenSSL API for most things is absolutely horrid. No self-respecting software engineer could have designed them. Objects which in reality share a sibling or parent-child relationship are implemented drastically differently, with dissimilar methods to work with them. Methods need to be used on objects in a certain unintuitive order for no apparent reason, or else things break.

The source code under the hood is terrible too. A huge lack of consistency. Operations are done in a very roundabout manner. Issues with one platform are solved by making things worse for all other platforms. Most things are not implemented particularly well. In fact the only thing OpenSSL is particularly good at is being ubiquitous.

Alternatives to OpenSSL

Now, several of the alternatives are much more nicely engineered and with saner APIs. However they all seem to fail basic quality in certain areas. For example, ASN.1 handling is generally written by people who don't know the ins and outs of data structures and best practice and algorithms. If you want good ASN.1, you'll need to have a database design expert to create it for you, not the engineers who are better at cryptography or simple API design. It should also be repeated that nice engineering doesn't equate secure, there's just so much collective messy knowledge which needs to be added.

So now enter the OpenBSD developers, who have a track record for meticulous programming and generally know what they're doing. They're looking to vastly cleanup OpenSSL. I've been waiting for something like this for years. They're going to take the best SSL/TLS library, and going to bring it up to decent engineering standards.

Fixing OpenSSL itself with the OpenSSL team and development structure is really not an option. Since it's a dumping ground, there's no true quality control as needed. Since its main aim is to be ubiquitous, the ancient platforms will inherently be dragging down the modern ones.

LibreSSL progress

So far, the OpenBSD fork of OpenSSL has deleted tons of code. This is crucial, as the more code there is, the more opportunity for bugs. Newbie programmers don't fully understand this point. Newbies think the more verbose some code is, the better. However, more code means more room for mistakes, and it also increases the code size. The more code used in a function, the harder it is to fully comprehend all of it. We humans have limits to how much data we can juggle in our heads. In order to ensure things can be fully conceptualized and analyzed, it has to be as short and as modular as possible. More library usage for similar techniques, and removing similar yet different copies of code littered throughout, as OpenBSD is aiming towards will ensure much higher levels of quality.

OpenBSD is also working to ensure LibreSSL does not contain the year 2038 problem, extending compatibility far into the future. Some of the random method order usage requirements are removed, making development less error prone with LibreSSL. LibreSSL in a very short time is becoming much more lean and more correct than OpenSSL ever was.

LibreSSL pitfalls

However, with the good, there is also bad. LibreSSL is aiming to be compatible with existing software using OpenSSL, which means brain damaged APIs will continue to exist. LibreSSL right now is also being modified to use OpenBSD specific functionality when OpenBSD's existing technology is more secure than what OpenSSL was doing. This means that for the time being, LibreSSL will only work correctly with OpenBSD. Since some functionality won't exist on other platforms, or that the functionality will exist, but not be nearly as secure as OpenBSD's implementation, potentially making ports which seem straight forward to actually be less secure than OpenSSL currently is.

Once I saw what OpenBSD was doing, I predicted to myself that there would be those looking to port LibreSSL to other platforms, and fail to account for all the security considerations. It seems I was right. In just a few short days since LibreSSL's announcement, I'm seeing a multitude of porting projects pop up all over for Linux, Windows, OS X, FreeBSD, and more, all by developers who may know some software engineering, but don't know the first thing about proper cryptographic implementations, or understanding the specific advantages of the OpenBSD implementations.

These porting implementations are failing many of the generic things listed above:

• Memory wiping is being optimized out.
• Random values are not being seeded with proper entropy.
• Random values contain noticeable patterns or bias.
• Bugs fixed in OpenSSL by switching from generic memory functions to OpenBSD's are being reintroduced by being switched back to generic or naive implementations of the OpenBSD memory functions.
• Constant-time functions aren't constant on the platforms being ported to.

The developers not understanding what they're doing is why LibreSSL was created in the first place. Unfortunately these porters are not security experts, and are creating or nabbing naive implementations from various sources. They're sometimes also grabbing source from OpenBSD itself or other security focused projects without realizing the source in question is being compiled with certain parameters or with special compilers to gain their security, and the implementation without the same build setup is hardly secure.

As with OpenSSH, in the future, the OpenBSD team will probably be providing a portable LibreSSL. But until then, avoid cheap knockoffs.

LibreSSL future

Will LibreSSL be successful? From what I've seen, it already is. You just may not have it available for your platform at the moment. The API is still crud, and as much as the OpenBSD team is cleaning things up, their code usually could go a bit farther in terms of brevity and clarity. However, the library will undoubtedly be superior to use compared to OpenSSL, and will serve as a much cleaner starting point to document how to create other implementations. Let's just hope the OpenBSD team is or will learn to be as knowledgeable regarding the SSL/TLS algorithms and gotchas as the OpenSSL team was.

Now before you start clamoring to start creating a high level implementation of SSL/TLS, or complaining that C is bad, and it should be written in Java. Consider that it's not even possible to program a secure SSL/TLS library in Java! (Source: Cryptography Engineering page 122).

Designing C++ functions to write/save to any storage mechanism

Problem

A common issue when dealing with a custom object or any kind of data is to create some sort of save functionality with it, perhaps writing some text or binary to a file. So what is the correct C++ method to allow an object to save its data anywhere?

An initial approach to allow some custom object to be able to save its data to a file is to create a member function like so:

void save(const char *filename);

While this is perfectly reasonable, what if I want something more advanced than that? Say I don't want the data to be saved as its own separate file, but would rather the data be written to some file that is already open, to a particular location within it? What if I'd rather save the data to a database? How about send the data over the network?

Naive Approach

When C++ programmers hear the initial set of requirements, they generally look to one of two solutions:

The first is to allow for a save function which can take an std::ostream, like so:

void save(std::ostream &stream);

C++ out of the box offers std::cout as an instance of an std::ostream which writes to the screen. C++ offers a derived class std::ofstream (std::fstream) which can save to files on disk. C++ also offers a derived class std::ostringstream which saves file to a C++ string.

With these options, you can display the data on the screen, save it to an actual file, or save it to a string, which you can then in turn save it wherever you want.

The next option programmers look to is to overload  std::basic_ostream::operator<< for the custom object. This way one can simply write:

mystream << myobject;

And then the object can be written to any C++ stream.

Either of these techniques pretty much work, but can be a bit annoying when you want a lot of flexibility and performance.

Say I wanted to save my object over the network, what do I do? I could save it to a string stream, grab the string, and then send that over the network, even though that seems a bit wasteful.

And for a similar case, say I have an already open file descriptor, and wish to save my object to it, do I also use a string stream as an intermediary?

Since C++ is extensible, one could actually create their own std::basic_streambuf derived class which works with file descriptors, and attach it to an std::ostream, which can then be used with anything that works with a stream for output. I'm not going to go into the details how to do that here, but The C++ Standard Library explains the general idea, and provides a working file descriptor streambuf example and shows how to use it with stream functions. You can also find some ready made implementations online with a bit of searching, and some compilers may even include a solution out of the box in their C++ extensions.

On UNIX systems, once you have a stream which works with file descriptors, you can now send data over the network, as sockets themselves are file descriptors. On Windows, you'll need a separate class which works with SOCKETs. Of course to turn a file descriptor streambuf into a SOCKET streambuf is trivial, and can probably be done with a few well crafted search and replace commands.

Now this may have solved the extra string overhead with file descriptors and networking, but what about if I want to save to a database? What about if I'm working with C's FILE *? Does one now have to implement a new wrapper for each of these (or pray the compiler offers an extension, or one can be found online)? The C++ stream library is actually a bit bloaty, and creating your own streambufs is somewhat annoying, especially if you want to do it right and allow for buffering. Many stream related library code you find online are also of poor quality. Surely there must be a better option, right?

Solution

If we look back at how C handles this problem, it uses function pointers, where the function doing the writing receives a callback to use for the actual writing, and the programmer using it can make the writing go anywhere. C++ of course includes this ability, and even takes it much further, in the form of function objects, and even further in C++ 2011.

Let's start with an example.

template<typename WriteFunction>
void world(WriteFunction func)
{
  //Do some stuff...
  //Do some more stuff...
  func("World", 5); //Write 5 characters via callback
  //Do some more stuff...
  unsigned char *data = ...;
  func(data, data_size); //Write some bytes
}

The template function above is expecting any function pointer which can be used to write data by passing it a pointer and a length. A proper signature would be something like the following:

void func(const void *data, size_t length);

Creating such a function is trivial. However, to be useful, writing needs to also include a destination of some sort, a device, a file, a database row, and so on, which makes function objects more powerful.

#include <cstdio>

class writer_file
{
  std::FILE *handle;
  public:
  writer_file(std::FILE *handle) : handle(handle) {}
  inline void operator()(const void *data, size_t length)
  {
    std::fwrite(data, 1, length, handle);
  }
};

Which can be used as follows:

world(writer_file(stdout));

Or perhaps:

std::FILE *fp = fopen("somefile.bin", "wb");
world(writer_file(fp));
std::close(fp);

As can be seen, our World function can write to any FILE *.

To allow any char-based stream to be written, the following function object will do the trick:

#include <ostream>

class writer_stream
{
  std::ostream *handle;
  public:
  writer_stream(std::ostream &handle) : handle(&handle) {}
  inline void operator()(const void *data, size_t length)
  {
    handle->write(reinterpret_cast<const char *>(data), length);
  }
};

You can call this with:

world(writer_stream(std::cout));

Or anything in the ostream family.

If for some reason we wanted to write to strings, it's easy to create a function object for them too, and we can use the string directly without involving a string stream.

#include <string>

class writer_string
{
  std::string *handle;
  public:
  writer_string(std::string &handle) : handle(&handle) {}
  inline void operator()(const void *data, size_t length)
  {
    handle->append(reinterpret_cast<const char *>(data), length);
  }
};

If you're worried about function objects being slow, then don't. Passing a function object like this to a template function has no overhead. The compiler is able to see a series of direct calls, and throws all the extraneous details away. It is as if the body of World is calling the write function to the handle passed to it directly. For more information, see Effective STL Item 46.

If you're wondering why developers forgo function pointers and function objects for situations like this, it is because C++ offers so much with its stream classes, which are also very extensible (and are often extended), they completely forget there are other options. The stream classes are also designed for formatting output, and working with all kinds of special objects. But if you just need raw writing or saving of data, the stream classes are overkill.

C++ 2011

Now C++ 2011 extends all this further in a multiple of ways.

std::bind()

First of all, C++ 2011 offers std::bind() which allows for creating function object adapters on the fly. std::bind() can take an unlimited amount of parameters. The first must be a function pointer of some sort, the next is optionally an object to work on in the case of a member function pointer, followed by the parameters to the function. These parameters can be hard coded by the caller, or bound via placeholders by the callee.

Here's how you would use std::bind() for using fwrite():

#include <functional>
world(std::bind(std::fwrite, std::placeholders::_1, 1, std::placeholders::_2, stdout));

Let us understand what is happening here. The function being called is std::fwrite(). It has 4 parameters. It's first parameter is the first parameter by the callee, denoted by std::placeholders::_1. The second parameter is being hard coded to 1 by the caller. The third parameter is the second parameter from the callee denoted by std::placeholders::_2. The fourth parameter is being hardcoded by the caller to stdout. It could be set to any FILE * as needed by the caller.

Now we'll see how this works with objects. To use with a stream, the basic approach is as follows:

world(std::bind(&std::ostream::write, &std::cout, std::placeholders::_1, std::placeholders::_2));

Note how we're turning a member function into a pointer, and we're also turning cout into a pointer so it can be passed as std::ostream::write's this pointer. The callee will pass its first and second parameters as the parameters to the stream write function. However, the above has a slight flaw, it will only work if writing is done with char * data. We can solve that with casting.

world(std::bind(reinterpret_cast<void (std::ostream::*)(const void *, size_t)>(&std::ostream::write), &std::cout, std::placeholders::_1, std::placeholders::_2));

Take a moment to notice that we're not just casting it to the needed function pointer, but as a member function pointer of std::ostream.

You might find doing this a bit more comfortable than using classical function objects. However, function objects still have their place, wherever functions do. Remember, functions are about re-usability, and some scenarios are complicated enough that you want to pull out a full blown function object.

For working with file descriptors, you might be tempted to do the following:

world(std::bind(::write, 1, std::placeholders::_1, std::placeholders::_2));

This here will have World write to file descriptor 1 - generally standard output. However this simple design is a mistake. Write can be interrupted by signals and needs to be resumed manually (by default, except on Solaris), among other issues, especially if the file descriptor is some kind of pipe or a socket. A proper write would be along the following lines:

#include <system_error>
#include <unistd.h>

class writer_fd
{
  int handle;
  public:
  writer_fd(int handle) : handle(handle) {}
  inline void operator()(const void *data, size_t length)
  {
    while (length)
    {
      ssize_t r = ::write(handle, data, length);
      if (r > 0) { data = static_cast<const char *>(data)+r; length -= r; }
      else if (!r) { break; }
      else if (errno != EINTR) { throw std::system_error(errno, std::system_category()); }
    }
  }
};

Lambda Functions

Now you might be wondering, why C++ 2011 stopped with std::bind(), what if the function body needs more than just a single function call that can be wrapped up in an adapter? That's where lambda functions come in.

world([&](const void *data, size_t length){ std::fwrite(data, 1, length, stdout); });
world([&](const void *data, size_t length){ std::cout.write(static_cast<const char *>(data), length); });

Note the ridiculous syntax. The [](){} combination signifies we are working with a lambda function. The [] receives a function scope, in this case &, which means that the function operates fully within its parent-scope, and has direct access to all its data. The rest you should already be well familiar with. You can change the stdout or the cout in the body of the lambda function to use your FILE * or ostream as necessary.

Let us look at an example of having our World function write directly to a buffer.

#include <cstring>

void *p = ...; //Point p at some buffer which has enough room to hold the contents needed to be written to it.
world([&](const void *data, size_t length){ std::memcpy(p, data, length); p = static_cast<char *>(p) + length; });

There's a very important point in this example. There is a pointer which is initialized to where writing should begin. Every time data is written, the pointer is incremented. This ensures that if World calls the passed write function multiple times, it will continue to work correctly. This was not needed for files above, as their write pointer increments automatically, or with std::string, where append always writes to the end, wherever it now is.

Be careful writing like this though, you must ensure in advance that your buffer is large enough, perhaps if your object has a way of reporting how much data the next call to its save or write function needs to generate. If it doesn't and you're winging it, something like the following is in order:

#include <stdexcept>

class writer_buffer
{
  void *handle, *limit;
  public:
  writer_buffer(void *handle, size_t limit) : handle(handle), limit(static_cast(handle)+limit) {}
  inline void operator()(const void *data, size_t length)
  {
    if ((static_cast<char *>(handle) + length) > limit) { throw std::out_of_range("writer_buffer"); }
    std::memcpy(handle, data, length);
    handle = static_cast<char *>(handle) + length;
  }
};

You can use it as follows:

#include <cstdlib>

size_t amount = 1024; //A nice number!
void *buffer = std::malloc(amount);
world(writer_buffer(buffer, amount));

Now an exception will be thrown if the callee tries to write more data than it should.

std::function

Lastly, C++ 2011 added the ability for more verbose type checking on function objects, and the ability to create the save/write function as a normal function as opposed to a template function. That ability is a general reusable function object facade, std::function.

To rewrite World to use it, we'd do as follows:

void world(std::function<void (const void *, size_t)> func)
{
  //Do some stuff...
  //Do some more stuff...
  func("World", 5); //Write 5 characters via callback
  //Do some more stuff...
  unsigned char *data = ...;
  func(data, data_size); //Write some bytes
}

With std::function, the type is now made explicit instead of being a template. It is anything which receives any kind of buffer and its length, and returns nothing. This can ensure that callers will always use a compatible function as intended by the library designer. For example, in our case, the caller only needs to ensure that data can be passed via a char * and an unsigned char *, based on how World uses the callback function. If World was now modified to also output an int *, less capable callers would now break. std::function can ensure that things are designed properly up front. With std::function, you can now also restructure your code to place various components in different compilation units if you so desire, although perhaps at a performance penalty.

Conclusion

To wrap up, you should now understand some features of C++ that are not as commonly used, or some new features of C++ 2011 that you may not be familiar with. You should now also have some ideas about generic code which should help you improve code you write.

Many examples above were given only with one methodology, although they can be implemented with some of the others. For practice, try doing this yourself. Also try applying these ideas to other kinds of storage mechanisms not covered here, doing so should now be rather trivial for you.

Remember, while this was done with a few standard examples and for writing, it can be extended to all handles Win32 offers, or for reading, or for anything else.

OAuth - A great way to cripple your API

Intro

 A few years ago, the big social networking sites were looking for a secure way to allow their users to safely use any and all untrusted software to access their own personal accounts. So that a user could have their account on one social networking site safely interact with their account on another social networking site. They also wanted to allow for users to be able to allow their accounts to interact safely with various untrusted web applications and web sites.

In order to safely allow untrusted software to access a user's account, a few points needed to be kept in mind.

• Untrusted software should not have access to a user's credentials, in case the software is compromised, passwords will not be stolen, as user's passwords are not stored by the software.
• Untrusted software should not have full access to a user's account, but only limited access as defined by the user. In the same vein, giving the software user's personal credentials will allow unlimited access to a user's account, which could be used maliciously.
• A user should be able to revoke the permission they granted to allow particular untrusted software to work, in case they turn malicious, despite the limited access they have.

A solution that was developed for this use-case was created - OAuth. OAuth's primary use-case is the one described above.

Implementation

OAuth is generally implemented in a fashion where the untrusted software is accessed by a user via a standard web browser. In order for a user to authorize that software, the software will redirect the user's browser to a page on the social networking site, with a couple of parameters sent to it. Now that the user is on his social networking site, the untrusted software no longer has control over what the user is doing, as the user left the untrusted software, allowing the user to safely log in to his social networking account. Then, the social networking site will see the parameters it was passed, and ask the user if he or she wants to authorize the software in question, and what kind of access to the user's account it should be given.

If and when the user has authorized the untrusted software, the social networking site can then report back to the untrusted software that it was granted a certain amount of access, and give it some credentials to use. These credentials are unique to the user and the software in question. The social networking site allows for a user to later on see a list of software authorized, and revoke the unique credentials given to any one of them, or modify the amount of access a particular set of credentials has.

Problems

There is no standard

Now above, I said OAuth is generally implemented in this fashion. I say generally, because unlike standard HTTP authentication schemes (Basic, Digest, and others), OAuth is a big grab bag of ideas which can be mixed and matched in an infinite amount of ways, and also allows for developers to make their own unique tweaks to their personal implementations.

With the standard HTTP authentication schemes, once a developer knows the protocol, and implemented it with one web site, that exact same knowledge can be reused to logging into any other web site that supports the standard. Likewise, software libraries can be made to handle HTTP authentication, and all a third party developer needs to do is specify to the library which credentials should be used, and then everything works as expected.

With OAuth, once a developer learns how to authenticate with it to one web site, it is likely that the same developer will need to relearn how to connect to every other web site using OAuth. This further means that every web site which supports OAuth needs to document exactly how it is implementing it, and what tweaks are in use. It also means that no library can be written which can simply support every OAuth implementation out there. This places a great burden on developers on both sides. It can also greatly increase frustration for less able users, when their favorite library works great with one site they support, but are unable to extend their software to another site, because their library lacks some aspect of OAuth for this new site, or a unique OAuth tweak on this new site renders the library incompatible.

Here are some choice quotes from the official RFC:

• However, as a rich and highly extensible framework with many optional components, on its own, this specification is likely to produce a wide range of non-interoperable implementations.
• This framework was designed with the clear expectation that future work will define prescriptive profiles and extensions necessary to achieve full web-scale interoperability.

Another issue is that while various standard HTTP authentication schemes have well understood security margins, OAuth is entirely variable. On one web site, its OAuth implementation may be secure, while on another, its implementation may be Swiss cheese. Even though a security consultant should generally look over any authorization implementation to ensure mistakes were not made, laymen can have a good understanding how reliable and secure their standardized authentication scheme is. A manager can ask their developers if some bullet points were adhered to, and then be reasonably confident in their security. Whereas with OAuth, the entire (complex) implementation needs to be reviewed from top to bottom by a top security professional to ensure it is secure. A manager has no adequate set of bullet points to discuss with their developers, as unique implementation details will drastically change the applicability of various points, with many important details still missing. At best, a manager can only get a false sense of security when OAuth is in use. The designers of OAuth respond to this point with a 71 page document of security issues that need to be dealt with!

What all this boils down to, is that OAuth is really just a set of ideas and recommendations on how to implement a unique authorization scheme. OAuth does not allow for interoperability as standards (usually) guarantee. Essentially, OAuth ensures that API authentication with your web site will be confusing, and will only work for the exact use-case for social networking sites described above.

Crippling Design - APUI

The common OAuth use-case described above is to allow for a user on one web site to allow their software to communicate for them with another web site. The workflow described above requires that a user navigate between web sites using their browser and manually enter their credentials and authorization to various aspects of their account as part of the overall software authorization process.

This means that OAuth (at least how it's normally implemented) only works between two web sites, with individual users, and that user intervention is required in order for software to authenticate with another web site. The fact that user intervention is required with manual input means that any API behind OAuth is not an API - Application Programming Interface, but an APUI - Application Programming User Interface, meaning user intervention is required for functionality. All in all, this cripples your API, or APUI as it should now properly be called.

Let us now focus on what cannot be properly done with OAuth in place:

• Third party software which is not part of a web site cannot interact with an OAuth APUI.
• One user in third party software cannot act on behalf of another user via an OAuth APUI.
• Third party software cannot run automated processes on an OAuth APUI.
• Organizations cannot ensure tight integration between various software and services they use.

Let us review an example case where a company launches a new online service for coordinating schedules and personal calendars between people. We'll call this new online service Calendar. The developers of Calendar create an APUI for it using OAuth, so third party software can integrate with Calendar.

As described above, a user needs to navigate between one web site and another in order to authorize software, with the two web sites sending each other information. What if one wants to integrate something which isn't a web site with Calendar? There's tons of desktop calendaring applications, Microsoft Outlook, Mozilla Lightning, Evolution, KOrganizer, Kontact, and more. If their developers want to integrate with the Calendar service, they need to embed a web browser in their applications now?

Furthermore, that may not even help, if the OAuth workflow requires information be sent to an existing web site, what web site is associated with a user's personal copy of software? Developers of that software are unlikely to create a web site with user accounts just for this integration. Even if they did, if the site goes down (temporarily), then the integration stops working. Even though, in theory, there should be no dependance of some extra web site between Calendar and the software trying to integrate with it.

Insecure

Also, as a security consideration, if an application does embed a web browser in it so it can authenticate with OAuth, the first requirement is no longer met - Untrusted software should not have access to a user's credentials. When using a standard web browser, once a user leaves the third party software's site and redirects to Calendar's site, the third party software cannot steal the information the user is entering. But when the third party software itself is the one which browses to Calendar's site, then it has access to everything the user is entering, including passwords.

Actually, this attack works on OAuth in every circumstance, including web site to web site, and I've actually used it in practice. A web site can embed a web browser via a Java Applet or similar, or have a web browser server side which presents the OAuth log in page to the user, but slightly modified to have all the data entered pass through the third party site. Therefore OAuth doesn't even fulfill its own primary security objective!

Incompatible with Enterprise

Next, once we go to the enterprise level, OAuth starts becoming much worse. Say a boss has his secretary manage his calendar for him, which is a very common scenario. In many OAuth setups, he cannot enter his Calendar credentials into the company-wide calendaring software running on a secure company server, which the secretary can then access. Rather, he would need to enter it directly in the browser the secretary uses, and stay there checking off many options. Also it is common that OAuth implementations are using security tokens which expire, meaning the boss will need to keep reentering his Calendar credentials again and again. Most bosses will just get fed up and give his secretary his credentials, especially if he's not physically near his secretary. It is also likely that the secretary will then write the password down. This gets compounded if multiple secretaries manage his schedule.

Now say an organization has software for which it would like to run background processes for all its users with Calendar. Perhaps every week it would like to automatically analyze the Calendar accounts of all department chiefs to find a good time for a weekly meeting. How would this work exactly? Every department chief would now have to go and enter their credentials into this software? Then do it again each time security tokens expire from past authentications?


Of course this is only the beginning. Since OAuth was designed for the likes of Twitter and Facebook which cater to individual personal accounts, the common implementations do not allow for hierarchical permissions as enterprise organizations need.

In enterprise infrastructure, software is already used which has well defined roles and access rights for ever single user. This infrastructure should be deciding who can do what, and to what level one user can act on behalf of another user. Once an organization purchases accounts for all their employees from Calendar, no employee should be able to turn off or limit what the enterprise management software can do with their Calendar account. The enterprise management software is the one who needs to make such decisions. This flies in the face of Untrusted software should not have full access to a user's account, but only limited access as defined by the user and A user should be able to revoke the permission they granted to allow particular untrusted software to work.

The amount of work involved to tightly integrate Calendar with existing infrastructure is also enormous from a user perspective. Every single account now has to have a user navigate across web pages and check all kinds of boxes until all the users are integrated. OAuth implementations generally forget to have administrator accounts which can do everything on behalf of other users in their organization.

It should be obvious at this point whether enterprise organizations would be willing to purchase accounts for their entire staff with Calendar, when integration into their existing infrastructure or new endeavors with the service will be difficult if not outright impossible. Imagine how much money Calendar sales now stand to lose from being unattractive to enterprise clients.

Repel third party developers

OAuth implementations also generally require that any software which uses their APUIs be preregistered in advance. This places extra burden on third party developers. The OAuth implementations also commonly require that the third party web site integrating be preregistered in advance, so you can say goodbye to your application being compatible with staging servers which may be at another URL. It also makes software much less attractive when one party sells software to another party, as every new client now needs to go through an application registration process for the URLs they plan to setup the software they are purchasing. Therefore third party developers are less likely to be interested in selling software which integrates with Calendar, as there's hassle involved with every single sale.

Recap

• OAuth is not a standard, but a set of ideas, which does not allow third party developers to reuse knowledge, and places extra documentation burden on implementers.
• OAuth as a whole has undefined security characteristics.
• OAuth doesn't properly fulfill its primary security objectives.
• OAuth doesn't work well outside social networking web site use-cases.
• OAuth services are unattractive to enterprise organizations looking to integrate such services into their infrastructure.
• OAuth services are less likely to have professional third parties sell software based upon them.

Solutions

If you're looking to implement authorization for your API, I recommend to sticking with well understood secure designs, such as HTTP Basic Authentication over SSL/TLS (or HTTP Digest Authentication).

In order to achieve a situation where users can securely authorize third party software, without giving over their personal credentials (passwords), I recommend that these services have a page where they can generate new credentials (keys) which the user can copy and paste. They can then name these keys themselves (avoiding application registration hassle), and set permissions upon them themselves. Since the user is the one initiating the key creation, and copying and pasting it themselves, they cannot fall prey to a man-in-the-middle attack where the third party software initiates the authorization process.

But remember the use-cases described here, and ensure that organizations have a way to access all user accounts company-wide, and without individual users being able to disable or limit that access.

Conclusion

If you want your service to be used by everyone out there, be well supported by third parties, and to have them create all kinds of interesting software with it, do not use OAuth.

Even the original social networking sites behind OAuth decided they really need other options for different use-cases, such as Twitter's xAuth, or Yahoo offering Direct OAuth, which turns the entire scheme into a more complicated version of HTTP Basic Authentication, with no added benefits. Perhaps the most damaging point against OAuth, is that the original designer behind it decided to remove his name from the specification, and is washing his hands clean of it.

I find it really amazing at how blind many big players are these days to all the problems with OAuth. When I first heard that IBM, a major enterprise player started offering services only accessible with typical utterly crippled OAuth, I was overcome with disbelief. Yet at the same time, I hear that they're wondering why they're not seeing the sales they used to with other services they offer, or compared to the competition.

I'm even more amazed to see other big companies throwing away their currently working authentication systems for OAuth. Followed by them wondering why many third party developers are not upgrading to support the new authentication scheme, and clients jumping ship to inferior services.

It seems many developers and managers out there simply don't get it. If you know anyone like that, show them this article.