Creating portable Linux binaries

For some, the idea of creating a portable Linux binary is somewhat elusive.

In this article, we will be discussing how to create a Linux binary for a specific architecture, that you will have great success running on a large variety of Linux distros. This includes current releases, somewhat old ones, and hopefully far into the future.

A common problem facing those looking to deploy proprietary software on Linux, or for those trying to supply binaries to a very large user-base which will not compile your software themselves, is how to offer one binary that fits most normal scenarios.

There are generally four naive approaches to solving this problem.

1. The developers set up a bunch of specific distros, and compile the software on each of them, and give out distro specific binaries. This makes sense at first, till you run into some trouble.
   • You have to juggle many live CDs or maintain a bunch of installed distros which is painful and time consuming.
   • You end up only offering support for the distros you have handy, and you will get quite a few users on a more exotic distro nagging you for support, or a different and incompatible version of a distro you're already supporting.
   • The compilers or other build utilities on some distros are too old for your modern software, and you need to build them elsewhere, or figure out how to back port modern software to that old distro.
   • Builds break when a user upgrades their system.
   • Users end up needing to install some non standard system libraries, increasing everyone's frustration.
2. The developers just statically link the binaries. This isn't always a legal option due to some licenses that may be involved. Binaries which are fully statically linked also in many instances exhibit incorrect behavior (more on this later).
3. The developers just compile the software on one system, and pray that it works for everyone else.
4. Compile with a really really old distro and hope it works everywhere else. However this succumbs to the last three problems outlined in naive approach #1, and in many cases the binaries produced won't work with modern distros.

Now there are plenty of companies that supply those portable Linux binaries. You find on their website downloads for say Linux i386, AMD64, and PPC. Somehow that i386 binary manages to run on every i386 system you've tested, Red Hat, Debian, SUSE, Ubuntu, Gentoo, and both old and modern versions at that. What is their secret sauce?

Now let us dive into all the important information and techniques to accomplish this worthy goal.

First thing you want to know is what exactly is your binary linked to anyway? For this, the handy ldd command comes in.

/tmp> ldd myapp
        linux-vdso.so.1 =>  (0x00007fff7a1ff000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1f8a765000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1f8a4e3000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1f8a2cc000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1f89f45000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1f8aaa9000)
/tmp>

The lines which are directed to a file are system libraries that you need to worry about. In this case, there's libstdc++, libm, libgcc, and libc. libc and libm are both part of (E)GLIBC, the C library that most Linux applications will be using. libstdc++ is GCC's C++ library. libgcc is GCC's implementation of some programming constructs that your program may be using, such as exception handling, and things like that.

In general (E)GLIBC is broken up into many sub libraries that your program may be linked against. Other notable examples are libdl for Dynamic Loading, libpthread for threading, librt for various real time functions, and a few others.

Your application will not run on a system unless all these dependencies are found, and are compatible. Therefore, versions of these also come into play. In general a newer minor version of a library will work, but not an older.

In order to find versions numbers, you want to use objdump. Here's an example with finding out what version of (E)GLIBC is needed:

/tmp> objdump -T myapp | grep GLIBC_
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ungetc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   __ctype_toupper_loc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fputc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 free
…
/tmp>

In this case, 2.3 is the highest version number. Therefore this binary needs (E)GLIBC 2.3 or higher on the system. Note, the version numbers have nothing to do with the version installed on your system, rather (E)GLIBC marks each function with the minimum version that contains it.

Of course all this applies to other libraries as well, particularly libgcc and libstdc++.

Now that we know a little bit about what we're doing, I'm going to present the first bit of secret sauce.

If you're using C++, link with -static-libstdc++ this will ensure that libstdc++ is linked statically, but won't link every other lib statically like -static would. You want libstdc++ linked statically, because it's safe to do so, some systems may be using an older version (or have none at all, in the case of some servers), or you want your binary to remain compatible if a new major version of libstdc++ comes out which is no longer backwards compatible. Note that even though libstdc++ is GPL'd, it also offers a linking exception that allows you to link against it and even statically link it in any application.

If you see that your binary needs libgcc, also use -static-libgcc for the same reasons given above. Also, GCC is GPL'd, and has the same linking exception as above. I once had the unfortunate scenario where I sold a client an application without libgcc statically linked, that used exceptions. On his old server, as long as everything went absolutely perfectly, the application ran fine, but if any issue occurred, instead of gracefully handling the issue, the application terminated immediately. Since his libgcc was too old, the application saw the throws, but none of the catches. Statically linking libgcc fixed this issue.

Now you might be thinking, hey what about statically linking (E)GLIBC? Let me warn you that doing so is a bad idea. Some features in (E)GLIBC will only work if the statically linked (E)GLIBC is the exact same version of (E)GLIBC installed on the system, making statically linking pointless, if not downright problematic. (E)GLIBC's libdl is quite notable in this regard, as well as several networking functions. (E)GLIBC is also licensed under LGPL. Which essentially means that if you give out the source to your application, then in most cases you can distribute statically linked binaries with it, but otherwise, not. Also, since 99% of the functions are marked as requiring extremely old versions of (E)GLIBC, statically linking is hardly necessary in most cases.

The next bit of the secret sauce is statically linking those non standard libs your application needs but nothing else.

You probably never learned in school how to selectively static link those libraries you want, but it is indeed possible. Before the list of libraries you wish to static link, place -Wl,-Bstatic and afterwards -Wl,-Bdynamic.

Say in my application I want to statically link libcurl and OpenSSL, but want to dynamically link zlib, and the rest of my libs, such as other parts of (E)GLIBC, I would use the following as my link flags:
gcc -o app *.o -static-libgcc -Wl,-Bstatic -lcurl -lssl -lcrypto -Wl,-Bdynamic -lz -ldl -lpthread -lrt

The next step is to ensure that your libraries pull in as few dependencies as possible. Here's the output from ldd on my libcurl.so:

        linux-vdso.so.1 =>  (0x00007fffbadff000)
        libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f84410a4000)
        libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f8440e7b000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f8440c6b000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f8440a1a000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8440812000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f84405d2000)
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8440374000)
        libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f843ff90000)
        librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f843fd75000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f843fb5e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f843f7d7000)
        libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f843f558000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f843f342000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f843f127000)
        libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f843ee66000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f843ec4a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f844157e000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f843e976000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f843e74c000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f843e548000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f843e33f000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f843e13a000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f843df36000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f843dd32000)
        libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f843db21000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f843d90f000)

This is quite unacceptable. Distros generally compile packages with everything enabled. Your application generally does not need everything a library has to offer. In the case of libcurl, you can compile it yourself, and disable the features you aren't using. In an example application, I only need HTTP and FTP support, so I could compile libcurl with very little, and now have this:

        linux-vdso.so.1 =>  (0x00007fffbadff000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8440812000)
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8440374000)
        libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f843ff90000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f843fb5e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f843f7d7000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f843f342000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f843ec4a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f844157e000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f843e13a000)

This is much more manageable. Refer to the documentation of your libraries in order to see how to compile them without  features you don't need.

If you're going to be compiling your own libraries, you probably want to set up a second system, virtual machine, or a chroot for building your customized library versions and applications, to ensure it doesn't conflict with your main system. Especially for the upcoming tip.

Secret sauce part 3, push your (E)GLIBC requirements down.

Let's look at an objdump on OpenSSL.

/tmp> objdump -T /usr/lib/libcrypto.so.0.9.8 | grep GLIBC_
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 chmod
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fileno
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __sysv_signal
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 printf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ftell
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getgid
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 shutdown
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 close
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 syslog
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ioctl
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 abort
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memchr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 gethostbyname
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fseek
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.7   __isoc99_sscanf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 openlog
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 exit
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strcasecmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 gettimeofday
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 setvbuf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 read
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strncmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 malloc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fopen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 setsockopt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sysconf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getpid
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fgets
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 geteuid
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 vfprintf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 closelog
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fputc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 times
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 free
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strlen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ferror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 opendir
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __xstat
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 listen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   __ctype_b_loc
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 __cxa_finalize
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 readdir
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 dlerror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sprintf
0000000000000000      DO *UND*  0000000000000000  GLIBC_2.2.5 stdin
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strrchr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 dlclose
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 poll
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getegid
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 gmtime_r
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strerror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sigaction
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strcat
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getsockopt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fputs
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 lseek
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strtol
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getsockname
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 connect
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memmove
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strchr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 socket
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fread
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __fxstat
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getenv
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __errno_location
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 qsort
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strncasecmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strcmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getuid
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   __ctype_tolower_loc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memcmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 feof
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fclose
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 dlopen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 recvfrom
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strncpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 dlsym
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 closedir
0000000000000000      DO *UND*  0000000000000000  GLIBC_2.2.5 stderr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fopen64
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sendto
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 bind
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fwrite
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 realloc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 perror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fprintf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 localtime
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 write
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 accept
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strtoul
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 open
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 time
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fflush
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getservbyname
/tmp>

It seems that OpenSSL would work on (E)GLIBC 2.3, except for one pesky function which needs 2.7+. This is a problem if I want to ship an application with this modern OpenSSL on say Red Hat Enterprise Linux 5 which comes with GLIBC 2.5, or say Debian Stable from ~4 years ago, which only has 2.4.

In this case OpenSSL is using a C99 version of sscanf(), but not actually by choice.

In /usr/include/stdio.h on (E)GLIBC 2.7+, you'll notice two blocks:

#if defined __USE_ISOC99 && !defined __USE_GNU \
    && (!defined __LDBL_COMPAT || !defined __REDIRECT) \
    && (defined __STRICT_ANSI__ || defined __USE_XOPEN2K)
# ifdef __REDIRECT
/* For strict ISO C99 or POSIX compliance disallow %as, %aS and %a[
   GNU extension which conflicts with valid %a followed by letter
   s, S or [.  */
extern int __REDIRECT (fscanf, (FILE *__restrict __stream,
        __const char *__restrict __format, ...),
           __isoc99_fscanf) __wur;
extern int __REDIRECT (scanf, (__const char *__restrict __format, ...),
           __isoc99_scanf) __wur;
extern int __REDIRECT_NTH (sscanf, (__const char *__restrict __s,
            __const char *__restrict __format, ...),
         __isoc99_sscanf);
# else
extern int __isoc99_fscanf (FILE *__restrict __stream,
          __const char *__restrict __format, ...) __wur;
extern int __isoc99_scanf (__const char *__restrict __format, ...) __wur;
extern int __isoc99_sscanf (__const char *__restrict __s,
          __const char *__restrict __format, ...) __THROW;
#  define fscanf __isoc99_fscanf
#  define scanf __isoc99_scanf
#  define sscanf __isoc99_sscanf
# endif
#endif

And

# if !defined __USE_GNU \
     && (!defined __LDBL_COMPAT || !defined __REDIRECT) \
     && (defined __STRICT_ANSI__ || defined __USE_XOPEN2K)
#  ifdef __REDIRECT
/* For strict ISO C99 or POSIX compliance disallow %as, %aS and %a[
   GNU extension which conflicts with valid %a followed by letter
   s, S or [.  */
extern int __REDIRECT (vfscanf,
           (FILE *__restrict __s,
      __const char *__restrict __format, _G_va_list __arg),
           __isoc99_vfscanf)
     __attribute__ ((__format__ (__scanf__, 2, 0))) __wur;
extern int __REDIRECT (vscanf, (__const char *__restrict __format,
        _G_va_list __arg), __isoc99_vscanf)
     __attribute__ ((__format__ (__scanf__, 1, 0))) __wur;
extern int __REDIRECT_NTH (vsscanf,
         (__const char *__restrict __s,
          __const char *__restrict __format,
          _G_va_list __arg), __isoc99_vsscanf)
     __attribute__ ((__format__ (__scanf__, 2, 0)));
#  else
extern int __isoc99_vfscanf (FILE *__restrict __s,
           __const char *__restrict __format,
           _G_va_list __arg) __wur;
extern int __isoc99_vscanf (__const char *__restrict __format,
          _G_va_list __arg) __wur;
extern int __isoc99_vsscanf (__const char *__restrict __s,
           __const char *__restrict __format,
           _G_va_list __arg) __THROW;
#   define vfscanf __isoc99_vfscanf
#   define vscanf __isoc99_vscanf
#   define vsscanf __isoc99_vsscanf
#  endif
# endif

These two blocks of code make fscanf(), scanf(), sscanf(), vfscanf(), vscanf(), and vsscanf() use special C99 versions. Since older applications were already compiled against C89 versions, (E)GLIBC doesn't want to potentially break them and change how an existing function works. So instead, a new set of functions were created which only exist in (E)GLIBC 2.7+, and (E)GLIBC by default will direct all calls to these functions to the proper C99 versions when compiling.

Now there are some defines you can set in your library code and application code to ensure it uses the old more backwards compatible versions, but getting the exact right combination of defines without breaking anything else can be tricky. It may also be tedious to modify a code-base you're not familiar with.

Therefore, I recommend just deleting these two blocks from your <stdio.h> on your build system. You want your build system to be able to build everything for backwards compatibility, right?

If you're recompiling libraries like OpenSSL which are designed for massive portability with all kinds of systems, odds are, they're not looking for C99 support in basic scanf() family functions anyway. If you do happen to need C99 scanf() support in your application, I recommend that you add it manually with a specialized lib, for maximum portability. You can easily find a bunch online.

The last scenario that you may encounter is that you happen to want to use a modern library function. For most libs you can just statically link them, but that won't work for (E)GLIBC. Since some functions depend on system support, or that custom versions don't perform as well as the built in system ones, you definitely want to use the built in ones if they're available. The question is, how to once the binary has already been compiled?

So for our final bit of secret sauce, dynamically load any modern functions that you want to use, and work around them, or disable some functionality if not present.

Remember libdl that we mentioned above? It offers dlopen() for opening system libraries, and dlsym() for finding out if certain functions are present or not, and retrieving a pointer to them.

I'm going to post a full example that you can look at and play with. In this example, we have a program which tries to figure out how big system pipes are. In this application, we are going to see how much data we can stuff in a pipe before we're told that the pipe is full, and the write would need to block.

Linux offers a function called pipe2() which has the crucial ability to create a pipe in non-blocking mode. If it doesn't exist, we can create it ourselves, but we prefer the built in one if possible.

#ifndef __linux__
#error This program is specifically designed for Linux, even though it works elsewhere
#endif


#include <stdio.h> //puts(), fputs(), printf(), fprintf(), stderr
#include <errno.h> //errno, perror(), EINTR, EAGAIN, EWOULDBLOCK
#include <fcntl.h> //fcntl(), F_SETFL, F_GETFL, O_NONBLOCK, F_SETFD, F_GETFD, FD_CLOEXEC, and for some: O_CLOEXEC
#include <dlfcn.h> //dlopen(), dlsym(), dlclose(), dlerror(), RTLD_LAZY
#include <unistd.h> //pipe() used in our implementation, write(), close()



//Lifted from: /usr/include/<arch>/bits/fcntl.h
#ifndef O_CLOEXEC
#define O_CLOEXEC       02000000        /* set close_on_exec */
#endif
//End lift


typedef int (*pipe2_t)(int [2], int);

//Implement the rather straight forward pipe2(), note: this function is of type: static pipe2_t
static int our_pipe2(int pipefd[2], int flags)
{
  int ret = pipe(pipefd);
  if (!ret) //Success, pipe created
  {
    //The built in pipe2() would not suffer from race conditions that the following code would succumb to in a threaded application
    if (flags & O_NONBLOCK)
    {
      fcntl(pipefd[0], F_SETFL, fcntl(pipefd[0], F_GETFL) | O_NONBLOCK);
      fcntl(pipefd[1], F_SETFL, fcntl(pipefd[1], F_GETFL) | O_NONBLOCK);
    }

    if (flags & O_CLOEXEC)
    {
      fcntl(pipefd[0], F_SETFD, fcntl(pipefd[0], F_GETFD) | FD_CLOEXEC);
      fcntl(pipefd[1], F_SETFD, fcntl(pipefd[1], F_GETFD) | FD_CLOEXEC);
    }
  }
  return(ret);
}

static pipe2_t pipe2 = our_pipe2; //pipe2() is initialized to our function


size_t pipe_size() //Manually determine the size of the system's pipe, for automatic, look up Linux specific F_GETPIPE_SZ
{
  //Create a union for using a pipe, so usage is a bit more logical
  union
  {
    int pipefd[2];
    struct
    {
      int read;
      int write;
    } side;
  } u;

  size_t amount = 0; //A pipe size of 0 signifies unknown

  if (!pipe2(u.pipefd, O_NONBLOCK)) //Note, here pipe2() is used
  {
    for (;;) //Write to a pipe in a loop, the final amount should be the size of the pipe
    {
      ssize_t w = write(u.side.write, &amount, sizeof(size_t)); //Write a size_t to the pipe

      if (w > 0) { amount += w; } //Success, add amount written and then loop
      else if (w == 0) //Pipe was closed, and we certainly didn't close it
      {
        perror("Pipe unexpectedly closed");
        amount = 0; //Reset to unknown, because an error occured
        break;
      }
      else /* Error occured trying to write */ if (errno != EINTR) //And it wasn't an interruption, so something that needs handling
      {
        if ((errno != EAGAIN) && (errno != EWOULDBLOCK)) //Failed to write to pipe - and it's nothing we'd fix
        {
          perror("Failed to write to pipe");
          amount = 0; //Reset to unknown, because an error occured
        }
        //Else, pipe is full, we're done!

        break; //In either case, we're done writing to the pipe
      }
      //Else If (errno == EINTR), we'd just loop and try again
    }
    close(u.side.read);
    close(u.side.write);
  }
  else { perror("Failed to create pipe"); }

  return(amount);
}

int main(const int argc, const char *const *const argv)
{
  void *so = dlopen("libc.so.6", RTLD_LAZY); //Open the C library
  if (so)
  {
    void *sym = dlsym(so, "pipe2"); //Grab the handle to pipe2() if it exists
    if (sym) //Success!
    {
      pipe2 = (pipe2_t)sym; //Use the built in one instead of ours
      puts("Using system's pipe2().");
    }
    else { puts("Using our pipe2()."); }
  }
  else
  {
    puts("Using our pipe2().");
    fprintf(stderr, "Could not open C library: %s\n", dlerror());
  }

  //Here's the real work
  size_t a = pipe_size();
  if (a) { printf("Pipe size is: %zu\n", a); }
  else { fputs("Could not determine pipe size.\n", stderr); }

  if (so) { dlclose(so); }
  return(0);
}

Here's how to compile and run it:

/tmp> gcc -Wall -o pipe_test pipe_test.c -ldl
/tmp> ./pipe_test
Using system's pipe2().
Pipe size is: 65536
/tmp>

Now pipe2() Was added to GLIBC in 2.9, yet this binary here according to objdump only needs (E)GLIBC 2.2.5+. Here's the output from an older system with GLIBC 2.7, using the exact same binary created on a newer system:

/tmp> ./pipe_test
Using our pipe2().
Pipe size is: 65536
/tmp>

Lastly, let me recap all the techniques we learned.

• Use ldd and objdump to see version requirements of binaries and libraries.
• Statically link compiler and language libraries, such as libgcc and libstdc++.
• Statically link selected libraries, while dynamically linking others.
• Compile selected libraries with as little needed functionality as possible.
• Pushing (E)GLIBC requirements down, by being wary of functions which have changed over time, and (E)GLIBC redirects calls to them in newly compiled programs by default.
• Pushing (E)GLIBC requirements down by not directly using new functions, and instead working around their presence.

Doing all this, you'll still need to make different builds for different operating systems, and different architectures like x86 and ARM, but at least you won't be forced to for all different distros and versions thereof.

One thing of note, it's possible to have Linux with different C libraries, and in those cases, you may as well be using a different Operating System. You'll be hard pressed to make complex programs compiled against one C library run on Linux which uses a different C library, where the needed one is not present. Thankfully though, all the mainstream desktop and server distros all use (E)GLIBC.

In any case, the techniques you've learned here can also be applied to other setups too.  (E)GLIBC was only focused on in this article because of its popularity and its many gotchas, but many other libraries that you may use, particularly video and audio libraries have similar issues as well.

Progression and Regression of Desktop User Interfaces

As this Gregorian year comes to a close, with various new interfaces out now, and some new ones on the horizon, I decided to recap my personal experiences with user interfaces on the desktop, and see what the future will bring.

When I was younger, there were a few desktop environments floating around, and I've seen a couple of them at school or a friend's house. But the first one I had on my own personal computer, and really played around with was Windows 3.

Windows 3 came with something called Program Manager. Here's what it looked like:




The basic idea was that you had a "start screen", where you had your various applications grouped by their general category. Within each of these "groups", you had shortcuts to the specific applications. Now certain apps like a calculator you only used in a small window, but most serious apps were only used maximized. If you wanted to switch to another running app, you either pressed the alt+tab keyboard shortcut to cycle through them, or you minimized everything, where you then saw a screen listing all the currently running applications.

Microsoft also shipped a very nice file manager, known as "File Manager", which alone made it useful to use Windows. It was rather primitive though, and various companies released various add-ons for it to greatly enhance its abilities. I particularly loved Wiz File Manager Pro.



Various companies also made add-ons for Program Manager, such as to allow nested groups, or shortcuts directly on the main screen outside of a group, or the ability to change the icons of selected groups. Microsoft would've probably built some of these add-ons in if it continued development of Program Manager.

Now not everyone used Windows all the time back then, but only selectively started it up when they wanted something from it. I personally did everything in DOS unless I wanted to use a particular Windows app, such as file management or painting, or copying and pasting stuff around. Using Windows all the time could be annoying as it slowed down some DOS apps, or made some of them not start at all due to lack of memory and other issues.

In the summer of 1995, Microsoft released Windows 4 to the world. It came bundled with MS-DOS 7, and provided a whole new user experience.


Now the back-most window, the "desktop", no longer contained a list of the running programs (in explorer.exe mode), but rather you could put your own shortcuts and groups and groups of groups there. Rather running programs would appear at the bottom of the screen in a "taskbar". The taskbar now also contained a clock, and a "start menu", to launch applications. Some always running applications which were meant to be background tasks appeared as a tiny icon next to the clock in an area known as a "tray".

This was a major step forward in usability. Now, no matter which application you were currently using, you could see all the ones that were currently running on the edge of your screen. You could also easily click on one of them to switch to it. You didn't need to minimize all to see them anymore. Thanks to the start menu, you could also easily launch all your existing programs without needing to minimize all back down to Program Manager. The clock always being visible was also a nice touch.

Now when this came out, I could appreciate these improvements, but at the same time, I also hated it. A lot of us were using 640x480 resolutions on 14" screens back then. Having something steal screen space was extremely annoying. Also with how little memory systems had back at the time (4 or 8 MB of RAM), you generally weren't running more than 2 or 3 applications at a time and could not really appreciate the benefits of having an always present taskbar. Some people played with taskbar auto-hide because of this.

The start menu was also a tad ridiculous. Lots of clicks were needed to get anywhere. The default appearance also had too many useless things on it.

Did anyone actually use help? Did most people launch things from documents? Microsoft released a nice collection of utilities called "PowerToys" which contained "TweakUI" which you could use to make things work closer to how you want.

The default group programs installed to within the start menu was quite messy though. Software installers would not organize their shortcuts into the groups that Windows came with, but each installed their apps into their own unique group. Having 50 submenus pop out was rather unwieldy, and I personally organized each app after I installed it. Grouping into "System Tools", "Games", "Internet Related Applications", and so on. It was annoying to manually do all this though, as when removing an app, you had to remove its group manually. On upgrades, one would also have to readjust things each time too.

Windows 4 also came with the well known Windows Explorer file manager to replace the older one. It was across the board better than the vanilla version of File Manager that shipped with Windows 3.

I personally started dual booting MS-DOS 6.22 + Windows for Workgroups 3.11 and Windows 95 (and later tri-booted with OS/2 Warp). Basically I used DOS and Windows for pretty much everything, and Windows 95 for those apps that required it. Although I managed to get most apps to work with Windows 3 using Win32s.

As I got a larger screen and more RAM though, I finally started to appreciate what Windows 4 offered, and started to use it almost exclusively. I still exited Windows into DOS 7 though for games that needed to use more RAM, or ran quicker that way on our dinky processors from back then.

Then Windows 4.10 / Internet Explorer 4 came out which offered a couple of improvements. First was "quick launch" which allowed you to put shortcuts directly on your taskbar. You could also make more than one taskbar and put all your shortcuts on it. I personally loved this feature, I put one taskbar on top of my screen, and loaded it with shortcuts to all my common applications, and had one on the bottom for classical use. Now I only had to dive into the start menu for applications I rarely used.

It also offered a feature called "active desktop" which made the background of the desktop not just an image, but a web page. I initially loved the feature, as I edited my own page, and stuck in an input line which I would use to launch a web browser to my favorite search engine at the time (which changed monthly) with my text already searched for. After a while active desktop got annoying though, as every time IE crashed, it disabled it, and you had to deal with extra error messages, and go turn it on manually.

By default this new version also made every Windows Explorer window have this huge sidebar stealing your precious screen space. Thankfully though, you could turn it off.

All in all though, as our CPUs got faster, RAM became cheaper, and large screens more available, this interface was simply fantastic. I stopped booting into other OSs, or exiting Windows itself.

Then Windows 5 came out for consumers, and UI wise, there weren't really any significant changes. The default look used these oversized bars and buttons on each window, but one could easily turn that off. The start menu got a bit of a rearrangement to now feature your most used programs up front, and various actions got pushed off to the side. Since I already put my most used programs on my quick launch on top, this start menu was a complete waste of space. Thankfully, it could also be turned off. I still kept using Windows 98 though, as I didn't see any need for this new Windows XP, and it was just a memory hog in comparison at the time.

What was more interesting to me however was that at work, all our machines ran Linux with GNOME and KDE. When I first started working there, they made me take a course on how to use Emacs, as every programmer needs a text editor. I was greatly annoyed by the thing however, where was my shift highlight with shift+delete and shift+insert or ctrl+x and ctrl+v cut and paste? Thankfully though I soon found gedit and kedit which was like edit/notepad/wordpad but for Linux.

Now I personally don't use a lot of windowsy type software often. My primary usage of a desktop consists of using a text editor, calculator, file manager, console/terminal/prompt, hex editor, paint, cd/dvd burner, and web browser. Only rarely do I launch anything else. Perhaps a PDF, CHM, or DJVU reader when I need to read something.

After using Linux with GNOME/KDE at work for a while, I started to bring more of my work home with me and wanted these things installed on my own personal computer. So dual booting Windows 98 + Linux was the way to go. I started trying to tweak my desktop a bit, and found that KDE was way more capable than GNOME, as were most of their similar apps that I was using. KDE basically offered me everything that I loved about Windows 98, but on steroids. KWrite/KATE was notepad/wordpad but on steroids. The syntax highlighting was absolutely superb. KCalc was a fine calc.exe replacement. Konqueror made Windows Explorer seem like a joke in comparison.

Konqueror offered network transparency, thumbnails of everything rather quickly, even of text files (no pesky thumbs.db either!). An info list view which was downright amazing:
This is a must have feature here. Too often are documents distributed under meaningless file names. With this, and many other small features, nothing else I've seen has even come close to Konqueror in terms of file management.

Konsole was just like my handy DOS Prompt, except with tab support, and better maximizing, and copy and paste support. KHexEdit was simply better than anything I had for free on Windows. KolourPaint is mspaint.exe with support for way more image formats. K3b was also head and shoulders above Easy CD Creator or Nero Burning ROM. For Web Browsers, I was already using Mozilla anyway on Windows, and had the same option on Linux too.

For the basic UI, not only did KDE have everything I liked about Windows, it came with an organized start menu. Which also popped out directly, instead from a "programs" submenu.

The taskbar was also enhanced that I could stick various applets on it. I could stick volume control directly on it. Not a button which popped out a slider, the sliders themselves could appear on the taskbar. Now for example, I could easily adjust my microphone volume directly, without popping up or clicking on anything extra. There was an eyedropper which you could just push to find out the HTML color of anything appearing on the screen - great for web developers. Another thing which I absolutely love, I can see all my removable devices listed directly on my taskbar. If I have a disk in my drive, I see an icon for that drive appearing directly on my taskbar, and I can browse it, burn to it, eject it, whatever. With this, everything I need is basically at my finger tips.

Before long I found myself using Linux/KDE all the time. On newer machines I started to dual boot Windows XP with Linux/KDE, so I could play the occasional Windows game when I wanted to, but for real work, I'd be using Linux.

Then KDE 4 comes out, and basically half the stuff I loved about KDE was removed. No longer is it Windows on steroids. Now KDE 4.0 was not intended for public consumption. Somehow all the distros except for Debian seemed to miss that. Everyone wants to blame KDE for miscommunicating this, but it's quite clear 4.0 was only for developers if you watched the KDE 4 release videos. Any responsible package maintainer with a brain in their skull should've also realized that 4.0 was not ready for prime time. Yet it seems the people managing most distros are idiots that just need the "latest" version of everything, ignoring if it's better or not, or even stable.

At the time, all these users were upset, and all started switching to GNOME. I don't know why anyone who truly loved the power KDE gave you would do that. If KDE 3 > GNOME 2 > KDE 4, why did people migrate to GNOME 2 when they could just not "upgrade" from KDE 3? It seems to me that people never really appreciated what KDE offers in the first place if they bothered moving to GNOME instead of keeping what was awesome.

Nowadays people tell me that KDE 4 has feature parity with KDE 3, but I have no idea what they're talking about. The Konqueror info list feature that I described above still doesn't seem to exist in KDE 4. You can no longer have applets directly on your taskbar. Now I have to click a button to pop up a list of my devices, and only then can I do something with them. No way to quickly glance to see what is currently plugged in. Konsole's tabs now stretch to the full width of your screen for some pointless reason. If you want to switch between tabs with your mouse, prepare for carpal tunnel syndrome. Who thought that icons should grow if they can? It's exactly like those idiotic theoretical professors who spout that CPU usage must be maximized at all times, and therefore use algorithms that schedule things for maximum power draw, despite that in normal cases performance does not improve by using these algorithms. I'd rather pack in more data if possible, having multiple columns of information instead of just huge icons.

KHexEdit has also been completely destroyed. No longer is the column count locked to hex (16). I can't imagine anyone who seriously uses a hex editor designed the new version. For some reason, KDE now also has to act like your mouse only has one button, and right click context menus vanished from all over the place. It's like the entire KDE development team got invaded by Mac and GNOME users who are too stupid to deal with anything other than a big button in the middle of the screen.

Over in the Windows world. Windows 6 (with 6.1 comically being consumerized as "Windows 7") came out with a bit of a revamp. The new start menu seems to fail basic quality assurance tests for anything other than default settings. Try to set things to offer a classic start menu, this is what you get:


If you use extra large icons for your grandparents, you also find that half of the Control Panel is now inaccessible. Ever since Bill Gates left, it seems Windows is going down the drain.

But hardly are problems solely for KDE and Windows, GNOME 3 is also a major step back according to what most people tell me. Many of these users are now migrating to XFCE. If you like GNOME 2, why are you migrating to something else for? And what is it with people trying to fix what isn't broken? If you want to offer an alternate interface, great, but why break or remove the one you already have?

Now a new version of Windows is coming out with a new interface being called "Metro". They should really be calling it "Retro". It's Windows 3 Program Manager with a bunch of those third party add-ons, with a more modern look to it. Gone is the Windows 4+ taskbar so you can see what was running, and easily switch applications via mouse. Now you'll need to press the equivalent of a minimize all to get to the actual desktop. Another type of minimize to get back to Program Manager to launch something else, or "start screen" as they're now calling it.

So say goodbye to all the usability and productivity advantages Windows 4 offered us, they want to take us back to the "dark ages" of modern computing. Sure a taskbar-less interface makes sense on handheld devices with tiny screens or low resolution, but on modern 19"+ screens? The old Windows desktop+taskbar in the upcoming version of Windows is now just another app in their Metro UI. So "Metro" apps won't appear on the classic taskbar, and "classic" applications won't appear on the Metro desktop where running applications are listed.

I'm amazed at how self destructive the entire market became over the last few years. I'm not even sure who to blame, but someone has to do something about it. It's nice to see that a small group of people took KDE 3.5, and are continuing to develop it, but they're rebranding everything with crosses everywhere and calling it "Trinity" desktop. Just what we need, to bring religious issues now into desktop environments. What next? The political desktop?

Reading in an entire file at once in C++, part 2

Last week I discussed 6 different methods on how to quickly get an entire file into a C++ string.

The conclusion was that the straight forward method was the ideal method, this was verified across several compilers. Since then, people asked me a number of questions.

What if I want to use a vector instead of a string, are the speeds different?
Forget C++ containers, what about directly into a manually allocated buffer?
What about copying into a buffer via mmap?
What do these various cases say about compilers or their libraries? Can this indicate what they're good or bad at compiling?

So to establish some baseline numbers, I created an application which reads the same files the same amount of times using the fastest method possible. It directly creates a buffer aligned to the partition's blocking factor, informs the OS to do the fastest possible sequential read, and then reads the file directly into the buffer. I also tested the suggested method to mmap a file, and copy the contents from there to an allocated buffer. The times achieved for these were 5 and 9 respectively. The latter is slower because of the extra memcpy() required, you want to read directly into your destination buffer whenever possible. The fastest time I now achieved should more or less be the fastest theoretical limit I'm able to get with my hardware. Both GCC and LLVM got the exact same times. I did not test with VC++ here as it doesn't support POSIX 2008.

Now regarding our 6 methods from last time, all of them except the Rdbuf method are possible with std::vector, since there is no std::vector based std::istream.

An interesting thing to note is that C++ string implementations vary greatly from implementation to implementation. Some offer optimizations for very small strings, some offer optimizations for frequent copies, by using reference counting. Some always ensure the string ends with a 0 byte so you can immediately pass them as a C string. In this latter case, operations which operate on strings as a range are rather quick, as the data is copied, then a 0 is appended. Whereas a loop which constantly pushes bytes on the back will have to needlessly set the extra trailing byte to 0 each time. Vector implementations on the other hand don't need to worry about a trailing 0 byte, and generally don't try to internally use all kinds of elaborate storage methods. So if std::vector works for you, you may want to use that.

Let's review times for the 5 applicable methods with our various compilers.

GCC 4.6 with a vector:

Method	Duration
C	23.5
C++	22.8
Iterator	73
Assign	81.8
Copy	68

Whereas with a string:

Method	Duration
C	24.5
C++	24.5
Iterator	64.5
Assign	68
Copy	63

We see that with a vector, the basic methods became a bit faster, but interestingly enough, the others got slower. However, which methods are superior to the others have remained the same.

Now for LLVM 3 with a vector:

Method	Duration
C	8
C++	8
Iterator	860
Assign	1328
Copy	930

Versus for string:

Method	Duration
C	7.5
C++	7.5
Iterator	110
Assign	102
Copy	97

With LLVM, everything is slower with a vector, and for the more complex solutions, much much slower. There's two interesting things we can see about LLVM though. For more straight forward logic, their compiler's optimizations are extremely smart. The speeds approach the theoretical best. I did some profiling on GCC and LLVM, as they're using the same C and C++ libraries, and found that in the straight C/C++ methods for my test program, GCC made 300 memory allocations, but LLVM made only 200. LLVM apparently is smart enough to see inside the various allocations, skip the ones that aren't needed, and place the data directly into the output buffer. But for complex code, LLVM's optimizations aren't that great. In the case of vectors and iterators, downright awful. Someone should file some bug reports with them.

Now for Visual C++ 2010 using vector:

Method	Duration
C	17.8
C++	18.7
Iterator	180.6
Assign	159.5
Copy	165.6

And string:

Method	Duration
C	16.5
C++	20.4
Iterator	224.4
Assign	222.8
Copy	320

We see here that the Copy method, which uses push_back() got a huge performance improvement. This seems to indicate that the STL implementation adds a 0 at the end of each operation, especially push_back(), instead of just when c_str() is called. Otherwise, string is faster.

It's also sad to see that GCC while winning all the cases where iterators were involved, was significantly slower in all the straight forward cases. This seems to indicate that GCC has the smartest optimizations, but fails to optimize well when the logic is straightforward. Someone should look into that.

It seems if you're trying to hold a collection of bytes, or whatever your wchar_t is, but don't care about the specialties of any particular container, as long as you don't push_back() a lot, string seems to be faster.

Finally, here's a table of all the compilers and methods I tested ordered by speed:

Method	Duration
POSIX	5
LLVM 3.0 s C/C++	7.5
LLVM 3.0 v C/C++	8
MMAP	9
VC 2010 s C	16.5
VC 2010 v C	17.8
VC 2005 s C	18.3
VC 2010 v C++	19.7
VC 2010 s C++	20.4
VC 2005 s C++	21
GCC 4.6 v C++	22.8
GCC 4.6 v C	23.5
VC 2005 v C	24
GCC 4.6 s C/C++	24.5
VC 2005 v C++	26
LLVM 3.0 s Rdbuf	31.5
GCC 4.6 s Rdbuf	32.5
GCC 4.6 s Copy	63
GCC 4.6 s Iterator	64.5
GCC 4.6 s Assign	68
GCC 4.6 v Copy	68
GCC 4.6 v Iterator	73
GCC 4.6 v Assign	81.8
LLVM 3.0 s Copy	97
LLVM 3.0 s Assign	102
LLVM 3.0 s Iterator	110
VC 2010 v Assign	159.5
VC 2010 v Copy	165.6
VC 2005 v Copy	172
VC 2010 s Rdbuf	176.2
VC 2010 v Iterator	180.6
VC 2005 s Rdbuf	199
VC 2005 s Iterator	209.3
VC 2005 s Assign	221
VC 2010 s Assign	222.8
VC 2010 s Iterator	224.4
VC 2010 s Copy	320
VC 2005 v Iterator	370
VC 2005 v Assign	378
VC 2005 s Copy	483.5
LLVM 3.0 v Iterator	860
LLVM 3.0 v Copy	930
LLVM 3.0 v Assign	1328

How to read in a file in C++

So here's a simple question, what is the correct way to read in a file completely in C++?

Various people have various solutions, those who use the C API, C++ API, or some variation of tricks with iterators and algorithms. Wondering which method is the fastest, I thought I might as well put the various options to the test, and the results were surprising.

First, let me propose an API that we'll be using for the function. We'll send a function a C string (char *) of a filename, and we'll get back a C++ string (std::string) of the file contents. If the file cannot be opened, we'll throw an error why that is so. Of course you're welcome to change these functions to receive and return whatever format you prefer, but this is the prototype we'll be operating on:

std::string get_file_contents(const char *filename);

Our first technique to consider is using the C API to read directly into a string. We'll open a file using fopen(), calculate the file size by seeking to the end, and then size a string appropriately. We'll read the contents into the string, and then return it.

#include <string>
#include <cstdio>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::FILE *fp = std::fopen(filename, "rb");
  if (fp)
  {
    std::string contents;
    std::fseek(fp, 0, SEEK_END);
    contents.resize(std::ftell(fp));
    std::rewind(fp);
    std::fread(&contents[0], 1, contents.size(), fp);
    std::fclose(fp);
    return(contents);
  }
  throw(errno);
}

I'm dubbing this technique "method C". This is more or less the technique of any proficient C++ programmer who prefers C style I/O would look like.

The next technique we'll review is basically the same idea, but using C++ streams instead.

#include <fstream>
#include <string>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::string contents;
    in.seekg(0, std::ios::end);
    contents.resize(in.tellg());
    in.seekg(0, std::ios::beg);
    in.read(&contents[0], contents.size());
    in.close();
    return(contents);
  }
  throw(errno);
}

I'm dubbing this technique "method C++". Again, more or less a straight forward C++ implementation based on the same principals as before.

The next technique people consider is using istreambuf_iterator. This iterator is designed for really fast iteration out of stream buffers (files) in C++.

#include <fstream>
#include <streambuf>
#include <string>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    return(std::string((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>()));
  }
  throw(errno);
}

This method is liked by many because of how little code is needed to implement it, and you can read a file directly into all sorts of containers, not just strings. The method was also popularized by the Effective STL book. I'm dubbing the technique "method iterator".

Now some have looked at the last technique, and felt it could be optimized further, since if the string has an idea in advance how big it needs to be, it will reallocate less. So the idea is to reserve the size of the string, then pull the data in.

#include <fstream>
#include <streambuf>
#include <string>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::string contents;
    in.seekg(0, std::ios::end);
    contents.reserve(in.tellg());
    in.seekg(0, std::ios::beg);
    contents.assign((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
    in.close();
    return(contents);
  }
  throw(errno);
}

I will call this technique "method assign", since it uses the string's assign function.

Some have questioned the previous function, as assign() in some implementations may very well replace the internal buffer, and therefore not benefit from reserving. Better to call push_back() instead, which will keep the existing buffer if no reallocation is needed.

#include <fstream>
#include <streambuf>
#include <string>
#include <algorithm>
#include <iterator>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::string contents;
    in.seekg(0, std::ios::end);
    contents.reserve(in.tellg());
    in.seekg(0, std::ios::beg);
    std::copy((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>(), std::back_inserter(contents));
    in.close();
    return(contents);
  }
  throw(errno);
}

Combining std::copy() and std::back_inserter(), we can achieve our goal. I'm labeling this technique "method copy".

Lastly, some want to try another approach entirely. C++ streams have some very fast copying to another stream via operator<< on their internal buffers. Therefore, we can copy directly into a string stream, and then return the string that string stream uses.

#include <fstream>
#include <sstream>
#include <string>
#include <cerrno>

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::ostringstream contents;
    contents << in.rdbuf();
    in.close();
    return(contents.str());
  }
  throw(errno);
}

We'll call this technique "method rdbuf".


Now which is the fastest method to use if all you actually want to do is read the file into a string and return it? The exact speeds in relation to each other may vary from one implementation to another, but the overall margins between the various techniques should be similar.

I conducted my tests with libstdc++ and GCC 4.6, what you see may vary from this.

I tested with multiple megabyte files, reading in one after another, and repeated the tests a dozen times and averaged the results.

Method	Duration
C	24.5
C++	24.5
Iterator	64.5
Assign	68
Copy	62.5
Rdbuf	32.5

Ordered by speed:

Method	Duration
C/C++	24.5
Rdbuf	32.5
Copy	62.5
Iterator	64.5
Assign	68

These results are rather interesting. There was no speed difference at all whether using the C or C++ API for reading a file. This should be obvious to us all, but yet many people strangely think that the C API has less overhead. The straight forward vanilla methods were also faster than anything involving iterators.

C++ stream to stream copying is really fast. It probably only took a bit longer than the vanilla method due to some reallocations needed. If you're doing disk file to disk file though, you probably want to consider this option, and go directly from in stream to out stream.

Using the istreambuf_iterator methods while popular and concise are actually rather slow. Sure they're faster than istream_iterators (with skipping turned off), but they can't compete with more direct methods.

A C++ string's internal assign() function, at least in libstdc++, seems to throw away the existing buffer (at the time of this writing), so reserving then assigning is rather useless. On the other hand, reading directly into a string, or a different container for that matter, isn't necessarily your most optimal solution where iterators are concerned. Using the external std::copy() function, along with back inserting after reservation is faster than straight up initialization. You might want to consider this method for inserting into some other containers. In fact, I found that std::copy() of istreambuf_iterators with back inserter into an std::deque to be faster than straight up initialization (81 vs 88.5), despite a Deque not being able to reserve room in advance (nor does such make sense with a Deque).

I also found this to be a cute way to get a file into a container backwards, despite a Deque being rather useless for working with file contents.

std::deque<char> contents;
std::copy((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>(), std::front_inserter(contents));

Now go out there and speed up your applications!

If there's any demand, I'll see about performing these tests with other C++ implementations.