Using Ecto changesets for JSON API request parameter validation

As Elixir developers, we typically use Ecto.Changeset to validate proposed database records changes in Elixir. It has comprehensive field validation capabilities and standardized error reporting, both of which we can customize.

In this post, I will share how you can use Ecto.Changeset beyond the database context, and use it as an API validation mechanism. Combine it with Gettext, and you get easy error translation for your APIs.

Regular Ecto

If youve worked with Elixir, then youve probably seen Ecto schemas and changesets like this:

@primary_key {:user_id, :binary_id, autogenerate: true}schema "users" do  field :username, :string, unique: true  field :password, :string, virtual: true, redact: true  field :password_hash, :string  field :first_name, :string  field :last_name, :string  field :is_banned, :boolean, default: false  field :is_deleted, :boolean, default: false  embeds_one :photo, Photo  timestamps()enddef changeset(user, attrs) do  user  |> cast(attrs, [:username, :first_name, :last_name, :password])  |> validate_required([:username, :first_name, :password])  |> shared_validations()end

This is a regular Ecto.Schema for a users table, with a changeset/2 function that receives the User struct and the proposed database record changes as attrs.

Passing valid data to changeset/2

When passing valid data to the changeset function changeset/2, it will return the tuple {:ok, %Ecto.Changeset{valid?: true, changes: changes, ...}}. Where changes is a map of validated database record changes, like:

%{first_name: "Jane", last_name: "Doe", username: "janedoe"}

Passing invalid data to changeset/2

When passing invalid data to the changeset/2 function, it will return the tuple {:error, %Ecto.Changeset{valid?: false, errors: errors, ...}}. Where errors is a keyword list, like:

errors: [  first_name: {"should be at least %{count} character(s)",    [count: 3, validation: :length, kind: :min, type: :string]},  last_name: {"should be at most %{count} character(s)",    [count: 20, validation: :length, kind: :max, type: :string]},  username: {"can't be blank", [validation: :required]}]

Which is a bit unreadable, but with the help of Ecto.Changeset.traverse_errors(changeset, &translate_error/1) we can translate these errors in a human-readable map, like:

%{  "first_name" => ["should be at least 3 character(s)"],  "last_name" => ["should be at most 20 character(s)"],  "username" => ["can't be blank"]}

Well be using the changes and errors fields later in this post, so lets keep them in mind.

When creating a Phoenix project, translate_error/1 is provided by default. If youve chosen to install Gettext (installed by default for a Phoenix app), you get easy language translation of these errors for free.

API validation

Using Ecto.Changeset for database record changes is nice, but lets take a look at API validation. Ill use a JSON-based API in the following examples, because thats what Im most familiar with.

Basic controller

Below is an example of a basic Phoenix controller, which creates a User and an EmailAddress, and verifies and updates an existing EmailConfirmation. Each action has a function call. These three function calls happen in succession, and each must succeed before another can be called.

That means we will want to validate the request parameters before we start any of the function calls; because if one succeeds, a later one may fail when it receives invalid data.

# API.AccountControlleraction_fallback(API.FallbackController)def create(conn, %{"account" => params}) do  with {:ok, email_confirmation} <- verify_email_confirmation(params),       {:ok, email_address} <- create_email_address(params),       {:ok, user} <- create_user(params) do    conn    |> put_status(:created)    |> render("create.json", user: user, email_address: email_address)  endenddef create(_conn, _params), do: {:error, :bad_request}

Lets see how we can validate the input parameters, before we call any of the functions.

Basic validation

We could add a pattern match in the controller arguments, and add some manual validation in private functions in the same controller file.

With the pattern match def create(conn, %{"email" => _, "code" => _, "username => _, "first_name" => _ "last_name" => _} = params) do we can guarantee all the fields that we need are present. Theres no validation of the values yet.

Because we lack an API validation library, we need to manually confirm the validity of each field in the controller:

• email must be a valid email address, e.g. [email protected]
• code must be a valid confirmation code, e.g. 123456
• username must be a valid string without spaces and special characters, e.g. johndoe
• first_name and last_name must be strings with a minimum length of 3 characters and a maximum length of 20 characters.

We could do this with a private function:

# API.AccountControllerdefp validate_params(%{"email" => email, "first_name" => first_name, ...}) do  with true <- is_email(email),       true <- String.length(first_name) >= 3,       true <- String.length(last_name) <= 20,       ... do    :ok  else    false -> :error  endend

But that gets dirty, fast.

Using Ecto

Instead of doing validations manually, lets create an Ecto.Schema and an accompanying changeset called AccountController.CreateAction that contains our validations, in the controller namespace:

# API.AccountController.CreateActionembedded_schema do  field(:email, :string)  field(:code, :string)  field(:username, :string)  field(:first_name, :string)  field(:last_name, :string)enddef changeset(attrs) do  %CreateAction{}  |> cast(attrs, [:email, :code, :first_name, :last_name])  |> validate_required([:email, :code, :first_name, :last_name])  |> validate_length(:code, is: 6)  |> validate_length(:username, min: 3, max: 10)  |> validate_length(:first_name, min: 3, max: 20)  |> validate_length(:last_name, min: 1, max: 20)  |> validate_format(:email_address, @email_regex)  |> validate_format(:username, @username_regex)  |> validate_format(:first_name, @name_regex)  |> validate_format(:last_name, @name_regex)  |> update_change(:email, &String.downcase/1)  |> update_change(:username, &String.downcase/1)enddef validate_params(params) do  case changeset(params) do    %Ecto.Changeset{valid?: false} = changeset ->       {:error, changeset}    %Ecto.Changeset{valid?: true, changes: changes} ->       {:ok, changes}  endend

Im going a bit overboard with the validations, to show the extent to which you can validate API fields and values. It can be very fine-grained.

In my own projects, I tend to abstract these validations into shared functions. For example, the first_name and last_name validations happen in both the CreateAction and in User schemas, so they share a separate validation function in User.

For example:

def validate_name(changeset, field) do  changeset  |> validate_format(field, @name_regex)  |> validate_length(field, min: 3, max: 20)end

Very nice.

Implementation

OK. Lets implement the validate_params/1 function of API.AccountController.CreateAction in the controller:

# API.AccountControllerdef create(conn, params) do  with {:ok, attrs} <- CreateAction.validate_params(params),       {:ok, email_confirmation} <- verify_email_confirmation(attrs),       {:ok, email_address} <- create_email_address(attrs),       {:ok, user} <- create_user(attrs) do    conn    |> put_status(:created)    |> render("create.json", user: user, email_address: email_address)  endend

Much cleaner. So what happens here?

We call CreateAction.validate_params/1 before any other function gets involved. validate_params/1 receives the request parameters as a map, and validates them using changeset/1, returning either {:ok, attrs} or {:error, changeset}.

If the request parameters are valid, then the Ecto.Changeset struct contains the valid?: true and changes: changes fields. changes is the map of validated request parameters that we want to pass to our subsequent function calls as {:ok, attrs}.

If the request parameters are invalid, then the Ecto.Changeset struct contains the valid?: false field, and we pass the Ecto.Changeset back to our controller function as {:error, changeset}, where it gets picked up by the FallbackController.

Error messages

So when the API request body contains invalid parameters, we receive {:error, %Ecto.Changeset{}}. To process this error, we need a FallbackController. Luckily, this is provided by default in a Phoenix project.

If youre missing FallbackController, then you can run one of the mix phx.gen tasks from https://hexdocs.pm/phoenix/Mix.Tasks.Phx.Gen.html and it will be generated for you.

Catching errors with FallbackController

The default FallbackController contains a fallback function like this:

# API.FallbackControllerdef call(conn, {:error, %Ecto.Changeset{} = changeset}) do  conn  |> put_status(:unprocessable_entity)  |> put_view(ChangesetView)  |> render("error.json", changeset: changeset)end

Whenever a function inside a controller returns {:error, %Ecto.Changeset{}}, it is caught by this fallback function inside FallbackController. The function then renders the changeset errors as a message, and returns the connection with a 422 HTTP status code (:unprocessable_entity).

For example, it returns error messages like this:

%{  "errors" => %{    "code" => ["should be 6 character(s)"],    "email_address" => ["has invalid format"]  }}

You can customize the error parsing with Ecto.Changeset.traverse_errors/2, but the default provided by Phoenix is a nice format for a frontend system to handle.

Translating error messages

If you have Gettext installed (which is installed by default in a Phoenix project), then you can add custom error translations for any language you need.

Since the error messages returned from Ecto.Changeset are always in a simple and specific format, like "is in valid" "can't be blank" and "should be at least 8 character(s)", we can easily add error translations for our API.

I wont dive into the details of Gettext, but the previous example of a rendered error could easily be translated into this, in Spanish :

%{  "errors" => %{    "code" => ["debe tener 6 caracter(es)"],    "email_address" => ["tiene un formato invlido"]  }}

Not sure if this translates correctly, because I don't speak much Spanish. But it's a nice feature to have, right?

I hope you learned something today, and that this post will help you build better APIs in Elixir. The next post will be about Dialyzer, and why you should (always) use it for development. Godspeed, alchemists!

More information

• Read about Ecto.Schema and embedded_schema here: https://hexdocs.pm/ecto/Ecto.Schema.html
• Read about Ecto.Changeset and error messages from changeset validations here: https://hexdocs.pm/ecto/Ecto.Changeset.html
• Read about FallbackController here: https://hexdocs.pm/phoenix/Phoenix.Controller.html#action_fallback/1
• Read about Plug HTTP status codes here: https://hexdocs.pm/plug/Plug.Conn.Status.html
• Read about Phoenix generators here: https://hexdocs.pm/phoenix/Mix.Tasks.Phx.Gen.html
• Read about Gettext here: https://hexdocs.pm/gettext/Gettext.html