Saturday, May 17, 2014

Protecting private keys

Web servers use private keys which they alone have in order to secure connections with users. Private keys must be protected at all costs.

In order to protect private keys on disk, one generally encrypts them with a password, which is then needed by the web server upon launch in order to decrypt and use it in memory. However, if measures aren't taken to secure the memory containing the private key, it can be stolen from there too, which would be catastrophic.

Normally, one doesn't need to worry about outsiders getting a hold of data from memory unless the attackers have direct access to the server itself. But bugs like Heartbleed allow remote users to grab random data from memory. Once data is in memory, the application and all the libraries it uses could divulge secret data if there is a buffer overflow lurking somewhere.

To protect against exploiting such bugs, one should ensure that buffer overflows do not have access to memory containing private data. The memory containing private keys and similar kinds of data should be protected, meaning nothing should be allowed to read from them, not even the web server itself.

Now obviously a program needs some access to a private key in order to work with it, so it can't just prevent all access from it. Rather, once a private key or similar data is loaded into memory, that memory should have its read permissions removed. When, and only when some activity needs to be performed with the private key, read permissions can be restored, the activity performed, and then read permissions revoked. This will ensure the rest of the application cannot access what it does not need, nor should be allowed to access.

On UNIX systems, one can use mprotect() to change the permission on a page of memory. On Windows, one can use VirtualProtect().

The above however has a crucial flaw - multi-threading. In a threaded application, all threads have access to data of other threads. So while one thread may be performing some critical private key related code, and allows read access for the moment, another thread can read it outside of the critical portion of code too. Therefore, even more isolation is needed.

To truly isolate the code that uses a private key and similar data, all the code that handles that stuff should be placed into its own process. The rest of the application can then request that well defined activities be performed via a pipe or other form of inter-process communication. This will also ensure that other kinds of bugs in the application, such as buffer overflows that allow arbitrary code execution cannot reestablish read access to the secret data.

On UNIX systems, one can use fork() to create a process which is still part of the same application. On all systems, the separate process can be a separate application with a well defined restrictive IPC API with limited and secured access by the web server.

No insecure library or silly bug in your web server should ever allow the application to divulge such secrets. If such services are not utilizing the techniques above, then they're just biding their time until the next Heartbleed.


henke37 said...

The most common implementation of the separate process idea is going to be simply outsourcing the ssl stuff entirely to a separate process.

It would ironically not protect against a vulnerability like the hearthbleed bug, but hopefully there will be no more such issues. It is the application code that is the normal danger and as such it is the code that needs the most distrust.

insane coder said...

Outsourcing all of SSL would be the wrong approach.

One needs to solely outsource the code which encrypts/decrypts with a private key and similar.

dreamer said...

This post makes perfect theoretical sense. However I am wondering what's the solution for servers/services that:

- Don't allow encrypted private keys (for instance, certain XMPP servers, postfix etc.)

- Operators, who, even if the software supports encrypted private keys, would need to take manual action when restarting services when/if they crash. This can be quite the operational overhead and may well pose a security concern, in terms of availability.

insane coder said...

Hi dreamer,

You're right that these issues prevent the keys to be encrypted on disk. Even so, if you are using unencrypted keys on disk, you don't necessarily want them to be freely accessible in memory, so things like Heartbleed, which don't even need file system access can grab them.

I don't know of any bullet proof solution for the encrypted key issue. What I see done is either multiple sysadmins who can unlock the keys as needed, and use multiple servers, so only some are offline and can be fixed more leisurely. Or use a key server to provide keys as need to the other servers, and move the problem to a different level.

insane coder said...

This technique is starting to be used:
TITUS (which is like stunnel):
OpenBSD's relayd:

justme said...

Hi. Just stumbled on this blogpost. I suppose one thing is protecting keys in memory. How about on disk? In a webfarm, these servers need to come up/restart without human intervention. I suppose a key-service might work to avoid packaging keys, but otherwise?

insane coder said...

Hello justme, see my comment above to dreamer which discusses this.

You may also want to invest in hardware specifically designed for storing keys which can self destruct in case your physical office is raided.

MBBS in Philippines said...

UV GULLAS COLLEGE OF MEDICINE is one of Top Medical College in Philippines in Cebu city. International students have the oppertunity to study medicine in phillipines at affordable cost and world class University. The college has successful alumni who have achieved well in the fields of law, business, politics, academe, medicine, sports and other endeavors. At University of the Visayas, we prepare students for a global competition.

Direct MBBS Admissions Open: 2020-21
Mobile No: +91 90329 55688
Apply Now:

augustwalker said...

Download the latest version of Garmin Update for Windows. Manage your GPS device with this official app from Garmin. Garmin Express is a tool developed by...

Hindifly said...

What is Bitcoin & How to Buy Bitcoin

Unknown said...

Bayzat is redefining the work life experience,
health insurance dubai
medical insurance dubai
making automated HR, payroll, employee benefits and insurance a possibility for all businesses

Author said...

its a great article !!!!!!!!! JobAlert247 A Job Alert Website

MovieRulz Best Movie Download Website

Digital Vishnu said...

This is incredibly useful information!! Excellent work. All is very fascinating to learn and simple to grasp. Thanks for sharing such great info. Keep Post These kinds of Articles in the future.

Digital Marketing Course in Coimbatore
Digital Marketing Course Training in Tirupur
Digital Marketing Course Training in Madurai
Digital Marketing Course Training in Theni
Digital Marketing Training in Coimbatore

MBBS in Philippines said...

Wisdom Overseasis authorized India's Exclusive Partner of Southwestern University PHINMA, the Philippines established its strong trust in the minds of all the Indian medical aspirants and their parents. Under the excellent leadership of the founder Director Mr. Thummala Ravikanth, Wisdom meritoriously won the hearts of thousands of future doctors and was praised as the “Top Medical Career Growth Specialists" among Overseas Medical Education Consultants in India.

Southwestern University PHINMAglobally recognized university in Cebu City, the Philippines facilitating educational service from 1946. With the sole aim of serving the world by providing an accessible, affordable, and high-quality education to all the local and foreign students. SWU PHINMA is undergoing continuous changes and shaping itself as the best leader with major improvements in academics, technology, and infrastructure also in improving the quality of student life.