Credit Cards and Personally identifiable information (PII) have compliance standards for their storage and transportation. PCI-DSS being one of the most commonly referenced compliance standards. There is an entire industry of payment processors like Stripe or Braintree who use making PCI-DSS compliance easier as a major selling point. These standards aren’t perfect, but they present a reasonable bar that companies can measure themselves by.

I see a class of data are not well covered by existing standards. I call them “Infrastructure Secrets”. Infrastructure Secrets are credentials or secrets that are commonly used to build or deploy applications and that they are often shared with third party services. Examples include:

• Github API keys
• AWS IAM identities
• Heroku API Tokens
• Rackspace Cloud API Keys
• SSH Private Keys

Many times systems using these secrets are running without direct human supervision, and the credentials they use are not locked down to single use cases. Many of these infrastructure providers do not have RBAC or limited use tokens. Even if the provider has an excellent system for limiting the scope of a credential, properly locked down tokens are still rarely used.

More than 4 years ago at Cloudkick we were dealing with these Infrastructure Secrets, including AWS API Keys and SSH keys to customer servers. I believe we took a paranoid approach to these securing these secrets. Cloudkick used all of the techniques that I outline bellow, and more.

Recent Events

MongoHQ is a startup providing MongoDB as a service. Last week they were hacked, based on a compromised password. CircleCI is another startup providing continuous integration and deployment. They happened to be using MongoHQ as their database provider. CircleCI stored many Infrastructure Secrets given to them by their customers. The diagram bellow outlines the steps in the attack:

This attack is a great example of privilege escalation across multiple providers and systems. Starting from just a compromised email password, the attacker escalated to SSH Keys, EC2 IAM credentials and more.

Threats

An in-depth threat model is application specific, but I want to outline some common threats that I see across many companies interacting with Infrastructure Secrets.

Bad Passwords

Individuals pick good passwords, but people pick bad ones. This is not going to change.

Support Tools

Support tools are critical components of a SaaS business, and most are built to allow an employee to do anything a customer can do — sometimes even with a direct impersonation feature so that the support team can see exactly what a customer would see. They tend to have poor security precautions or logging, and at the same time time access to every customer’s information.

Databases

Databases are backed up, they are left on employee hard disks, and in the case of CircleCI, even hosted with 3rd-parties. Newer NoSQL data stores tend to have simple access controls, and often companies use a single set of credentials for all access.

Server Compromise

Applications are being developed as quickly as possible using a relatively small set of server technologies. For example in January, there was an exploit against Ruby on Rails that allowed remote code execution. If an attacker were to uncover similar exploits in the future, attacking someone who stores infrastructure secrets could be much more lucrative that other targets.

Defensive Layers for Infrastructure Secrets

Defense in depth is a common approach to information security. I view security as a series of white picket fences. Jumping over a single fence might be easy, but jumping over 40 isn’t. If a system is partially compromised, having side effects that alert you to abnormal behaviors is also important.

Encryption using Keyczar

Don’t type the words AES, use Keyczar. Keyczar is a series of libraries built to have a simple API and sane defaults. The best cryptography algorithm will always change, but Keyczar provides file formats and mechanisms for changing these defaults over the life time of an application. To support data of various lengths the Keyczar Sessions API should generally be used. At Cloudkick we used Django and developed a KeyczarField that overrode models.TextField serialization to the database. We could then set the fields as if it was a normal ORM field in Django. If a backend service wanted to use the secrets, they had to explicitly decrypt them.

Isolated Services with specific APIs

If you use the Keyczar Sessions API you can put the public keys on all servers, but only put the private keys on specific backend servers. This can let a user update a credential from your web server, but only a specific backend service can view the decrypted value.

These backend servers should be on isolated networks and only provide exact operations over their communication channel.

For example, if you are storing SSH Keys to deploy code:

• DO NOT: Have a service with an exported API like run_command(host, command).
• DO: Have a service with a specific operation like: deploy_project_to_host(host, project)

In the event of a web server being compromised the attacker can only force more deploys to happen — this still might be bad, but it is one more white picket fence for them to jump over.

Notifications on special events

For Cloudkick, in addition to our normal logging when an employee activated impersonation, we would send an additional email to our root@ alias. The alias got a few emails every day, but an attacker abusing it would of been quickly noticed.

Multi-Factor Authentication

Multi-Factor authentication is a critical and cheap method to protect against the most common threat: bad passwords. For Cloudkick we used YubiKeys extensively. You don’t need an expensive RSA SecureID system, today there are many more options including standards compliant HOTP/TOTP or startups like Duo Security.

Be Paranoid

The ideas I outlined above are a random collection. Everything from Firewalls, Log collection, patching, secure software designs and more are needed. You can’t protect against all possible threats, like the NSA tapping Google shows, but a reasonable set of white picket fences can stop many threats.

OSWAP have created several guides and recommendations for common pitfalls of web applications. But I have not seen any content covering these types of Infrastructure Secrets. I believe a standard for storing, transporting and using Infrastructure Secrets is needed, and would love to see one evolve.