Database Encryption with the Phoenix Auth Generators

Written by Gus Workman
Published on 2023-02-13

In today’s digital age, users expect a high level of privacy when it comes to their personal information. It’s no secret that data breaches and leaks are becoming increasingly common, which makes it more important than ever to protect user data from unauthorized access. One effective way to do this is by encrypting the sensitive data stored in your database.

By encrypting data with a library like Cloak, you can prevent a data leak from exposing sensitive user information, and ensure that your users’ personal data remains private and secure.

In this blog post, we’ll dive into how to integrate the Cloak database encryption library into a Phoenix application, including integration with the standard Phoenix authentication generators.

Getting Started

I’m currently using Elixir 1.14.2 on OTP 25. I have the latest Phoenix generator installed as of the time of writing, which is 1.7.0-rc.2. To begin, I’m going to generate a new Phoenix app and then run the phx.gen.auth command as shown below. I’m using SQLite3 here in order to reduce local dependencies - but the default PostgreSQL will work just the same!

mix phx.new secure_app --database sqlite3
mix phx.gen.auth Accounts User users

After this step, you can check out your default home page at http://localhost:4000 and notice the register and log in links at the top left corner. Feel free to try registration, email verification, password reset, etc to get a feel for the experience that the authentication generator creates. As a reminder, you can also simulate verifying your email by checking the inbox at http://localhost:4000/dev/mailbox.

If we take a look inside the database, we can take a peek at the tables and the data it contains:

➜ sqlite3 secure_app_dev.db

SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.

sqlite> .tables
schema_migrations  users              users_tokens

sqlite> select * from users;
1|me@example.com|$2b$12$e/oTc3/PLYR60MRCN70lW.k8yqPYqgF0JZ0azhT.QpM3WuWeKgPLS|2023-02-12T02:23:21|2023-02-12T02:22:21|2023-02-12T02:23:21

This all looks pretty standard. We have a users table which stores an ID, email, hashed password and some timestamps. You can also see that this is the case by checking out the migration for the authentication tables:

defmodule SecureApp.Repo.Migrations.CreateUsersAuthTables do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :email, :string, null: false, collate: :nocase
      add :hashed_password, :string, null: false
      add :confirmed_at, :naive_datetime
      timestamps()
    end

    create unique_index(:users, [:email])

    create table(:users_tokens) do
      add :user_id, references(:users, on_delete: :delete_all), null: false
      add :token, :binary, null: false, size: 32
      add :context, :string, null: false
      add :sent_to, :string
      timestamps(updated_at: false)
    end

    create index(:users_tokens, [:user_id])
    create unique_index(:users_tokens, [:context, :token])
  end
end

We want to ensure our users’ privacy. In this case, we are going to consider the user email to be a sensitive field. If our server gets compromised, we don’t want the attacker to be able to read our database full of user emails. So how do we achieve that? Encrypt the email column!

Setup and Configuration of Cloak

To encrypt our database columns, we are going to use the excellent cloak and cloak_ecto libraries.

Let’s add them to our dependencies in mix.exs:

  defp deps do
    [
      # ...
      {:cloak, "~> 1.1.2"},
      {:cloak_ecto, "~> 1.2.0"}
    ]
  end

Next, we need to define and configure a vault for Cloak. The definition of this module is simple, since it mostly invokes the use of the Cloak.Vault macro, but behind the scenes it is the module performs encryption and decrytion of your data as it moves between your app and the database. The definition of the vault is as below - be sure to change the module name and the opt_app field to reflect your app’s name!

defmodule SecureApp.Vault do
  use Cloak.Vault, otp_app: :secure_app
end

Next, we need to supply this module with configuration options in dev.exs. There are a few keys of note. First, you can change the JSON library to your library of choice by setting the json_library key. The default is Jason. The second configuration option is the ciphers value. You could configure multiple cipher algorithms here, but we are only going to use the Cloak.Ciphers.AES.GCM implementation. For this cipher, we can also supply a keyword list of options with the tag, key and iv_length keys set. The tag is an identify to help the library detect which key data was encrypted with. The key is a secret value that is used to encrypt the data. Keep the key value safe, as leaking the key will allow anyone to decrypt your data. Finally, the iv_length is a parameter of the encyption algorith - the standard value recommended by the library author is 12.

config :secure_app, SecureApp.Vault,
  json_library: Jason,
  ciphers: [
    default: {Cloak.Ciphers.AES.GCM,
    tag: "AES.GCM.V1",
    key: Base.decode64!("your-key-here"),
    iv_length: 12}
  ]

You can see above that the key parameter is expecting a base64 encoded key value. Unfortunately, "your-key-here" is neither a valid base64 string nor would it be a secure key even if it was. So, let’s generate a new key. We can do so within iex:

iex(1)> 32 |> :crypto.strong_rand_bytes() |> Base.encode64
"0xO0pwiFpvutsAvmASgYJR9q.......54SY9r+M="

Fortunately, the above snippet of code is a very easy way to generate a strong key. I redacted some of the middle characters of the key - please be sure to generate a key for yourself, do not copy the one shown here!

Now that we have a key, just paste the string value into the config options for the SecureApp.Vault module.

The last step for the Cloak module is to start it in the supervision tree. In application.ex, I’ve added the following:

def start(_type, _args) do
  children = [
    # ...
    # Start the Vault
    SecureApp.Vault,
    # ...
  ]

  # ...
end

Now the Cloak Vault itself is set up - but we have one last step before completing the setup and configuration. That is to define our encrypted data types, similar to how we created the Vault module. There are a number of them available, corresponding to numerical data types, dates and times, as well as lists. Check out the Cloak Ecto docs to learn more about the supported data types. For this purpose, we want to encrypt our binaries (strings), so we will use the Cloak.Ecto.Binary module.

defmodule SecureApp.Encrypted.Binary do
  use Cloak.Ecto.Binary, vault: SecureApp.Vault
end

In addition, we want to be able to securely query the email field - for purposes such as log in a user from email and password, and checking that the email provided during registration is unique. To do this, we also need a hash field. Some common hashes are for example SHA256, HMAC, PBKDF2, etc. Although PBKDF2 is stronger, we are going to use HMAC in this tutorial because it has no extra dependencies. To use it in the project, add one more module as shown below:

defmodule SecureApp.Hashed.HMAC do
  use Cloak.Ecto.HMAC, otp_app: :secure_app
end

And this module has some simple configuration options, I will add them here below. In this application, I will use a secret key generated the same way as above. The key helps prevent attackers from using a rainbow table to easily recover the original contents of the hashed field.

config :secure_app, SecureApp.Hashed.HMAC,
  algorithm: :sha512,
  secret: "NGbyAd0gQgiIBfZLHiws.......XGwWSuyWp70w="

And that’s almost all there is to the configuration - now we can start encrypting our database fields!

Encrypting Authentication Tables with Cloak Ecto

The next part of the process is to implement the encryption on the sensitive columns of our authentication tables - in this case the email column of the users table.

To start, we need to re-define the data type in the migration and the schema for the users. In the migration, I will set the data type to :binary, because the encrypted data is stored in a raw binary format. For the schema, I will change the email field to the SecureApp.Encrypted.Binary data type that we defined above. Simple!

In the migration, change the email field to the following two fields:

create table(:users) do
  add :email, :binary, null: false
  add :email_hashed, :binary, null: false
  # ...
end

And in the user schema:

schema "users" do
  field :email, SecureApp.Encrypted.Binary
  field :email_hashed, SecureApp.Hashed.HMAC
  # ...
end

Next, we need to make sure that the email_hashed field is set when our changesets are invoked. To do so, let’s create the function, maybe_put_hashed_email/1, which will copy the email value to the email_hashed value if it is not null.

defp maybe_put_hashed_email(changeset) do
  email = get_field(changeset, :email)

  if email do
    put_change(changeset, :email_hashed, email |> String.downcase())
  else
    changeset
  end
end

Now, we want to make sure that this function is called in the validate_email/2 pipeline:

defp validate_email(changeset, opts) do
  changeset
  |> validate_required([:email])
  |> validate_format(:email, ~r/^[^\s]+@[^\s]+$/, message: "must have the @ sign and no spaces")
  |> validate_length(:email, max: 160)
  |> maybe_put_hashed_email() # <- added this line here
  |> maybe_validate_unique_email(opts)
end

And finally, we want to ensure that the maybe_validate_unique_email/2 function is using the email_hashed field instead of the email field, since the encrypted binary is not searchable:

defp maybe_validate_unique_email(changeset, opts) do
  if Keyword.get(opts, :validate_email, true) do
    changeset
    |> unsafe_validate_unique(:email_hashed, SecureApp.Repo)
    |> unique_constraint(:email_hashed)
  else
    changeset
  end
end

Okay so at this point, registering for a new account should work. However, I get an error flash message, Invalid email or password, after testing it out. Oddly enough, the user does exist in the database, however.

The reason for the error is that the LiveView event handlers are creating the user record in the database. After that succeeds, it submits the form to the login route, so the user can be logged in immediately after registration. The code for the login routes can be found in user_session_controller.ex.

By inspecting the controller code, we can see that the error message above is shown to the user when the call to Accounts.get_user_by_email_and_password/2 fails. By looking at it, we can notice that the query is trying to lookup the user by the email field. However, this is not searchable since it is encrypted - instead we need to change it to email_hashed, as below:

def get_user_by_email_and_password(email, password)
    when is_binary(email) and is_binary(password) do
  user = Repo.get_by(User, email_hashed: email)
  if User.valid_password?(user, password), do: user
end

We should also update the get_user_by_email/1 function just above it to use the same email_hashed field:

def get_user_by_email(email) when is_binary(email) do
  Repo.get_by(User, email_hashed: email)
end

Great! So if you go back through the registration and login process, it works without issue! We’re done, right? … Not quite, unfortunately.

Fixing the Tests

The phx.gen.auth also generates valid tests. Since we just changed implementation details of the authentication system, all of the public APIs should still be the same. So we can run the tests to ensure we haven’t broken anything:

mix test

...

Finished in 1.1 seconds (0.8s async, 0.3s sync)
128 tests, 107 failures

😬

The good news is that most of the problem is related to the configuration - if you remember, we only set up the Cloak key and cipher in dev.exs. So, I will copy the config over to test.exs, and generate new keys for the AES and HMAC configuration. While we’re at it, we might as well configure the application for production as well.

For a production deployment, we typically want to set the encryption keys as secrets or environment variables, depending on your hosting solution. You especially do not want to commit the secrets as a part of the code base. To read the configuration from the environment at application start, I will add the following configuration code in runtime.exs, in the :prod block:

if config_env() == :prod do
  # ...

  # read the encryption key parameters from environment
  encryption_key =
    System.get_env("ENCRYPTION_KEY") ||
      raise """
      environment variable ENCRYPTION_KEY is missing.
      You can generate one by running: 32 |> :crypto.strong_rand_bytes() |> Base.encode64
      in IEX
      """

  hmac_secret =
    System.get_env("HMAC_SECRET") ||
      raise """
      environment variable HMAC_SECRET is missing.
      You can generate one by running: 32 |> :crypto.strong_rand_bytes() |> Base.encode64
      in IEX
      """

  # configure the vault encryption method and parameters
  config :secure_app, SecureApp.Vault,
    json_library: Jason,
    ciphers: [
      default:
        {Cloak.Ciphers.AES.GCM,
         tag: "AES.GCM.V1", key: Base.decode64!(encryption_key), iv_length: 12}
    ]

  config :secure_app, SecureApp.Hashed.HMAC,
    algorithm: :sha512,
    secret: hmac_secret

  # ...
end

Now all we need to do is generate some new encryption keys and set the ENCRYPTION_KEY and HMAC_SECRET in our server’s secrets or environment variables. Reminder: do not store the plaintext value of these secrets within your repository!

After setting the proper test configuration, we can run mix test again:

mix test

...

Finished in 1.7 seconds (0.6s async, 1.1s sync)
128 tests, 12 failures

That’s much better at least! Still some work to get the tests working though. We will cover the rest in Part 2.

Stay tuned!