Managing environment variables with SecretsFoundry

I wrote about the different ways of managing environment variables earlier in my post here. To make it easy for us to deal with configuration in our startup at Truefoundry, we wrote a small tool called SecretsFoundry that has really made it quite seamless for all application teams to maintain their configurations in Git. We thought it might be useful for other developer teams and hence decided to open-source it.

Before I go into the details, it will be good to understand whats the problem SecretsFoundry solves. Every application has some non-sensitive and sensitive configuration variable which need to be provided to the application when its running. For the non-sensitive variables, people tend to put the variables in a file and then load the variables into the application using libraries like dotenv. For non-sensitive variables, people either store the values in some secret managers like AWS SecretManager, Hashicorp Vault and then write application code to pull the secrets from the store. The other approach is to have some external system infuse the variables from the secretstore into the application environment — in which case the domain of env variables becomes more of a devops responsibility and developers lose control of it — leading to more bugs and harder debugging when issues happen.

SecretsFoundry tries to solve the issues above by doing the following:

All sensitive and non-sensitive keys can live in one file.

For non-sensitive variables, you can put the variables directly. For sensitive variables, we put the path in the secretstore as the value of those variables. This way, we tell secretsfoundry how to fetch those values. An example of such a file will be:

.env file

NODE_ENV = development
HOST = localhost
DB_NAME = example_app_db
DB_USER = ${aws-secret:/development/example_app/DB_USER}
DB_PASSWORD = ${aws-secret:/development/example_app/DB_PASSWORD}

In the example above, the actual DB_USER and DB_PASSWORD are stored in AWS Secrets Manager. Developers can mention the path in the .env file and secretsfoundry will fetch it for you.

No application specific code to fetch env variables

Secretsfoundry works by infusing the actual values in the app environment before it starts up rather than within the application. This has two advantages:

1. No dependencies in the application and we don’t need to add libraries across the multitude of different languages.
2. In case secretsfoundry is not able to find a certain env variable, secretsfoundry itself errors out and provides an early unhealthy signal to all deployment systems like Kubernetes. Otherwise its on the application to do the validation and handle failures.

Support for multiple secret managers

SecretsFoundry integrates with AWS Parameter Store, AWS S3, AWS SecretsManager and Hashicorp Vault for secrets management. Adding a new secret store is quite easy and we do plan to extend support for GCP and Azure Vault in the future.

Using SecretsFoundry

The way to use secretsfoundry is:

secretsfoundry run -c "node start.js"

or if you have multiple commands in your startup script:

secretsfoundry run -s "node run_migration_script && node start.js"

When we do secretsfoundry run, it looks for a .env file in that directory and prints the values of the variables. If there is a -c or -s argument, it infuses them into the environment of that application. So if you consider the .env file we wrote earlier, running secretsfoundry run in that directory will output

secretsfoundry run

NODE_ENV = development
HOST = localhost
DB_NAME = example_app_db
DB_USER = admin
DB_PASSWORD = password

We can have a small sample application in start.js which looks something like below:

secretsfoundry run -c “node start.js”

SecretsFoundry can also load up any other .env.<stage> file using

secretsfoundry run — stage=<stage>

SecretsFoundry can pick up config files from another directory using

secretsfoundry run -p <Path to the config directory, that holds the .env files.>

It can also take input path to a file which can be in .env / json / yaml format and output the resolved variables to a different file. We use this for integrating with various Kubernetes systems which I will write about in another blog.

secretsfoundry run -i <Input file containing variables (.env/json/yaml)> -o <output_file>

SecretsFoundry needs credentials to get the parameters from the SecretStore — so you will have to provide those variables manually to your environment.

Installation and Docs

You can download secretsfoundry using npm or yarn (https://www.npmjs.com/package/secretsfoundry)

npm install -g secretsfoundry

SecretsFoundry resides in this Github repository and its docs can be found at https://truefoundry.gitbook.io/secretsfoundry.

Give SecretsFoundry a try and let us know your experience. We have been using it internally quite extensively with AWS Parameter Store — however the Hashicorp Vault integration hasn’t been tested well. Give it a try and let us know if you find any bugs or have any feature requests.

Originally Published on Medium