24 Mar 2020 · Software Engineering

    Test-Driven APIs with Phoenix and Elixir

    12 min read
    Contents

    Introduction

    Starting with a new technology can be difficult. It usually takes some time to understand the components and the way they work together. In this tutorial, we’re going to create an API, using Test Driven Development (TDD) to guide us through its implementation. You’ll learn how the feedback provided by our test helps save us time. It does so by showing us clues on every fail and giving us a better understanding of how things work.

    Prerequisites

    First of all, let’s make sure we have all the components we need:

    Check your Elixir version to ensure it matches the one we’re using:

    → elixir --version
    Elixir 1.1.1
    

    Getting Started

    We will start by thinking about what we want from our implementation. If we simply jump into generators, things will probably work, but we will have a lot of useless code in our codebase, and no understanding of the layers. Since we want to create an API, we do not need any HTML views or stylesheets. That’s a good place to start:

    mix phoenix.new watchlist --no-html --no-brunch
    

    Make sure to reply with yes (Y) to install the dependencies:

    Fetch and install dependencies? [Yn] Y
    

    We don’t need a database right now, but we will create one, since we are going to use it at a later stage. We’ll use Phoenix’s default database, PostgreSQL.

    mix ecto.create
    

    This will compile the project and test our connection to the database. A failure of ecto.create looks as follows:

    ** (Mix) The database for Watchlist.Repo couldn't be created, reason given: psql: could not connect to server: Connection refused
    

    Sometimes when the OS crashes, a Postgres lock file remains on the file system, making Postgres assume that there is a connection available:

    rm /usr/local/var/postgres/postmaster.pid
    

    Start (or restart) your Postgres server, and you’ll be good to go:

    → mix ecto.create
    The database for Watchlist.Repo has been created.
    

    Now everything looks fine. We can start thinking about our first feature.

    Developing a Feature: Listing Movies

    It’s time to start thinking about the feature. We should avoid dealing with the implementation details as much as possible.

    Adding an API Endpoint

    What do we know so far? We’ll return a JSON response, which means that we need a JSON library. We need Poison. Let’s start by adding the following dependencies to mix.exs:

     defp deps do
       [{:phoenix, "~> 1.0.4"},
        {:phoenix_ecto, "~> 1.1"},
        {:postgrex, ">= 0.0.0"},
        {:cowboy, "~> 1.0"},
        {:poison, "~> 1.5"}]
     end

    Next, we’ll run mix deps.get to get those dependencies.

    It’s time to create an integration test. Create a test/integration directory and a file named listing_movies_test.exs inside it.

    Listing movies requires a get request for a movies resource. We need a HTTP GET request to a ‘/movies’ URI to return some content and a 200 status code.

    We’ll start with a simple:

    defmodule ListingMoviesIntegrationTest do
      use ExUnit.Case, async: true
    
      test 'listing movies' do
        response = conn(:get, '/movies')
        assert response.status == 200
      end
    end

    Here’s how this will work: we’ll add this test and run it. After running the test, we will make changes every time it fails until we’ve added just enough code to make it pass. We’ll use mix test test/integration/listing_movies_test.exs to run the test. The test returns the following error message:

    ** (CompileError) test/integration/listing_movies_test.exs:5: function conn/2 undefined
    

    We need Plug. Let’s add it to test/integration/listing_movies_test.exs:

    use Plug.Test

    When we run it again, we’ll get the following output:

      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:5
         ** (FunctionClauseError) no function clause matching in URI.parse/1
         stacktrace:
           (elixir) lib/uri.ex:292: URI.parse('/movies')
           (plug) lib/plug/adapters/test/conn.ex:10: Plug.Adapters.Test.Conn.conn/4
           test/integration/listing_movies_test.exs:6
    

    The problem here is the single quotes. Elixir provides double quoted strings. Single quotes are for char lists. Let’s fix this:

    response = conn(:get, "/movies")
      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:5
         Assertion with == failed
         code: response.status() == 200
         lhs:  nil
         rhs:  200
         stacktrace:
           test/integration/listing_movies_test.exs:7
    

    We don’t have a status code because we are creating a connection, but we’re not making a call to an endpoint with it. Let’s add one. We’ll need to call a router, connect to it, and get a response:

    defmodule ListingMoviesIntegrationTest do
      use ExUnit.Case, async: true
      use Plug.Test
      alias Watchlist.Router
    
      @opts Router.init([])
      test 'listing movies' do
        conn = conn(:get, "/movies")
        response = Router.call(conn, @opts)
        assert response.status == 200
      end
    end
       1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:7
         ** (Phoenix.Router.NoRouteError) no route found for GET /movies (Watchlist.Router)
         stacktrace:
           (watchlist) web/router.ex:1: Watchlist.Router.match/4
           (watchlist) web/router.ex:1: Watchlist.Router.do_call/2
           test/integration/listing_movies_test.exs:9
    

    The test fails because we don’t have a route. We’re making progress. Let’s open web/router.ex and add our route:

    scope "/", Watchlist do
      pipe_through :api
    
      get "/movies", MovieController, :index
    end

    We’ll run the test again even though we know it is going to fail. In this case, we’ll get the following output:

       1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:7
         ** (Plug.Conn.WrapperError) ** (UndefinedFunctionError) undefined function: Watchlist.MovieController.init/1 (module Watchlist.MovieController is not available)
         stacktrace:
           Watchlist.MovieController.init(:index)
           (watchlist) web/router.ex:1: anonymous fn/1 in Watchlist.Router.match/4
           (watchlist) lib/phoenix/router.ex:255: Watchlist.Router.dispatch/2
           (watchlist) web/router.ex:1: Watchlist.Router.do_call/2
           test/integration/listing_movies_test.exs:9
    

    So, we need to create a controller. Let’s create a movie_controller.ex file on the web/controllers directory:

    defmodule Watchlist.MovieController do
      use Watchlist.Web, :controller
    
      def index(conn, _params) do
        render conn, movies: []
      end
    end
      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:7
         ** (Plug.Conn.WrapperError) ** (UndefinedFunctionError) undefined function: Watchlist.MovieView.render/2 (module Watchlist.MovieView is not available)
    

    Our action needs a view. Creating a movie_view.ex under web/views will fix that:

    defmodule Watchlist.MovieView do
      use Watchlist.Web, :view
    
      def render("index.json", %{movies: movies}) do
        movies
      end
    end

    Let’s run it again:

    Compiled web/views/movie_view.ex
    Generated watchlist app
    .
    
    Finished in 0.2 seconds (0.2s on load, 0.01s on tests)
    1 test, 0 failures
    

    Our first test has passed. Let’s add an assertion for the body to make sure we are on the right path: assert response.resp_body == "[]"

    The test passes. Now, let’s try to consume our resource from bash, run the server with mix phoenix.server, and use curl to retrieve the movies:

    → curl localhost:4000/movies
    []
    

    Adding a Model

    It works — our empty list is being returned. Now, we need to add a movie model to persist our movies. Again, we’ll use TDD to drive how our model is going to look. We could read the model straight away, but adding it to our test first will give us some time to consider what we want it to look like. Let’s give it a name and a rating, and assert its return:

    defmodule ListingMoviesIntegrationTest do
      use ExUnit.Case, async: true
      use Plug.Test
      alias Watchlist.Router
    
      @opts Router.init([])
      test 'listing movies' do
        movie = %Movie{name: "Back to the future", rating: 5}
                |> Repo.insert!
    
        conn = conn(:get, "/movies")
        response = Router.call(conn, @opts)
    
        assert response.status == 200
        assert response.resp_body == movie
      end
    end
    ** (CompileError) test/integration/listing_movies_test.exs:8: Movie.__struct__/0 is undefined, cannot expand struct Movie
        (elixir) src/elixir_map.erl:58: :elixir_map.translate_struct/4
    

    Structure does not exist yet. Let’s create it using the generator:

    → mix phoenix.gen.model Movie movies name rating:integer
    * creating priv/repo/migrations/20151204182719_create_movie.exs
    * creating web/models/movie.ex
    * creating test/models/movie_test.exs
    
    Remember to update your repository by running migrations:
    
        $ mix ecto.migrate
    

    Now, migrate and take a look at test/model/movie_test.exs the generator created for us:

    defmodule Watchlist.MovieTest do
      use Watchlist.ModelCase
    
      alias Watchlist.Movie
    
      @valid_attrs %{name: "some content", rating: 42}
      @invalid_attrs %{}
    
      test "changeset with valid attributes" do
        changeset = Movie.changeset(%Movie{}, @valid_attrs)
        assert changeset.valid?
      end
    
      test "changeset with invalid attributes" do
        changeset = Movie.changeset(%Movie{}, @invalid_attrs)
        refute changeset.valid?
      end
    end

    Let’s run this test:

    → mix test test/models/movie_test.exs
    ..
    
    Finished in 0.2 seconds (0.2s on load, 0.00s on tests)
    2 tests, 0 failures
    

    When we run the integration test again, we’ll get the following error:

    ** (CompileError) test/integration/listing_movies_test.exs:8: Movie.__struct__/0 is undefined, cannot expand struct Movie
        (elixir) src/elixir_map.erl:58: :elixir_map.translate_struct/4
    

    We’ve created our structure, but we haven’t added it to our test. Let’s add it:

    alias Watchlist.Movie

    Let’s run the test again:

      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:8
         ** (UndefinedFunctionError) undefined function: Repo.insert!/1 (module Repo is not available)
         stacktrace:
           Repo.insert!(%Watchlist.Movie{__meta__: #Ecto.Schema.Metadata<:built>, id: nil, inserted_at: nil, name: "Back to the future", rating: 5, updated_at: nil})
           test/integration/listing_movies_test.exs:10
    

    The structure is here, but we can’t insert it without using Repo.

    alias Watchlist.Repo
      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:9
         Assertion with == failed
         code: response.resp_body() == movie
         lhs:  "[]"
         rhs:  %Watchlist.Movie{__meta__: #Ecto.Schema.Metadata<:built>, id: nil, inserted_at: nil, name: "Back to the future", rating: 5, updated_at: nil}
         stacktrace:
           test/integration/listing_movies_test.exs:17
    

    There are still some things left to fix. First of all, we are returning a string, and we should be returning a list of movies. Let’s fix that in web/movie_controller.ex:

    defmodule Watchlist.MovieController do
      use Watchlist.Web, :controller
      alias Watchlist.Movie
    
      def index(conn, _params) do
        movies = Repo.all(Movie)
        render conn, movies: movies
      end
    end
      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:9
         ** (Plug.Conn.WrapperError) ** (Poison.EncodeError) unable to encode value: {nil, "movies"}
    

    This means that we are trying to encode metadata for the client, and Poison won’t allow us to do this by default. We can solve this by implementing Poison.Encoder in our model.

    defmodule Watchlist.Movie do
      use Watchlist.Web, :model
    
      schema "movies" do
        field :name, :string
        field :rating, :integer
    
        timestamps
      end
    
      @required_fields ~w(name rating)
      @optional_fields ~w()
    
      @doc """
      Creates a changeset based on the model and params.
    
      If no params are provided, an invalid changeset is returned
      with no validation performed.
      """
      def changeset(model, params \\ :empty) do
        model
        |> cast(params, @required_fields, @optional_fields)
      end
    
      defimpl Poison.Encoder, for: Watchlist.Movie do
        def encode(movie, _options) do
          movie
          |> Map.from_struct
          |> Map.drop([:__meta__, :__struct__])
          |> Poison.encode!
        end
      end
    end

    Let’s run the test again:

       1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:9
         Assertion with == failed
         code: response.resp_body() == movie
         lhs:  "[{\"updated_at\":\"2015-12-07T17:08:24Z\",\"rating\":5,\"name\":\"Back to the future\",\"inserted_at\":\"2015-12-07T17:08:24Z\",\"id\":91}]"
         rhs:  %Watchlist.Movie{__meta__: #Ecto.Schema.Metadata<:loaded>, id: 91, inserted_at: #Ecto.DateTime<2015-12-07T17:08:24Z>, name: "Back to the future", rating: 5,
                updated_at: #Ecto.DateTime<2015-12-07T17:08:24Z>}
    

    Notice we get a JSON response, but it fails. Our expectation is not encoded. Let’s encode the model we’ve just created in our integration test:

    |> Repo.insert!
    |> Poison.encode!

    The output now looks as follows:

      1) test listing movies (ListingMoviesIntegrationTest)
         test/integration/listing_movies_test.exs:9
         Assertion with == failed
         code: response.resp_body() == movie
         lhs:  "[{\"updated_at\":\"2015-12-07T17:11:27Z\",\"rating\":5,\"name\":\"Back to the future\",\"inserted_at\":\"2015-12-07T17:11:27Z\",\"id\":92}]"
         rhs:  "{\"updated_at\":\"2015-12-07T17:11:27Z\",\"rating\":5,\"name\":\"Back to the future\",\"inserted_at\":\"2015-12-07T17:11:27Z\",\"id\":92}"
    

    We’re comparing a list with a single movie. Let’s fix that:

    defmodule ListingMoviesIntegrationTest do
      use ExUnit.Case, async: true
      use Plug.Test
      alias Watchlist.Router
      alias Watchlist.Movie
      alias Watchlist.Repo
    
      @opts Router.init([])
      test 'listing movies' do
        %Movie{name: "Back to the future", rating: 5} |> Repo.insert!
        movies = Repo.all(Movie)
                 |> Poison.encode!
    
        conn = conn(:get, "/movies")
        response = Router.call(conn, @opts)
    
        assert response.status == 200
        assert response.resp_body == movies
      end
    end
    .
    
    Finished in 0.8 seconds (0.6s on load, 0.1s on tests)
    1 test, 0 failures
    

    Let’s try using it with Curl:

    → curl localhost:4000/movies
    [{"updated_at":"2015-12-04T21:46:08Z","rating":5,"name":"Ilha das flores","inserted_at":"2015-12-04T21:46:08Z","id":1}]
    

    Our resource is done. Let’s run all of our specs to make sure everything is working before going further:

    → mix test
    
    
      1) test renders 404.json (Watchlist.ErrorViewTest)
         test/views/error_view_test.exs:7
         Assertion with == failed
         code: render(Watchlist.ErrorView, "404.json", []) == %{"errors" => %{"detail" => "Page not found"}}
         lhs:  %{errors: %{detail: "Page not found"}}
         rhs:  %{"errors" => %{"detail" => "Page not found"}}
         stacktrace:
           test/views/error_view_test.exs:8
    
    
    
      2) test render 500.json (Watchlist.ErrorViewTest)
         test/views/error_view_test.exs:12
         Assertion with == failed
         code: render(Watchlist.ErrorView, "500.json", []) == %{"errors" => %{"detail" => "Server internal error"}}
         lhs:  %{errors: %{detail: "Server internal error"}}
         rhs:  %{"errors" => %{"detail" => "Server internal error"}}
         stacktrace:
           test/views/error_view_test.exs:13
    
    
    
      3) test render any other (Watchlist.ErrorViewTest)
         test/views/error_view_test.exs:17
         Assertion with == failed
         code: render(Watchlist.ErrorView, "505.json", []) == %{"errors" => %{"detail" => "Server internal error"}}
         lhs:  %{errors: %{detail: "Server internal error"}}
         rhs:  %{"errors" => %{"detail" => "Server internal error"}}
         stacktrace:
           test/views/error_view_test.exs:18
    
    ...
    
    Finished in 0.3 seconds (0.3s on load, 0.06s on tests)
    6 tests, 3 failures
    

    When the –no-html option was implemented, it was made with this syntax. It’s already been fixed on master, and it should be part of the next Phoenix release. For now, we can fix this ourselves.

    defmodule Watchlist.ErrorViewTest do
      use Watchlist.ConnCase, async: true
    
      # Bring render/3 and render_to_string/3 for testing custom views
      import Phoenix.View
    
      test "renders 404.json" do
        assert render(Watchlist.ErrorView, "404.json", []) ==
               %{errors: %{detail: "Page not found"}}
      end
    
      test "render 500.json" do
        assert render(Watchlist.ErrorView, "500.json", []) ==
               %{errors: %{detail: "Server internal error"}}
      end
    
      test "render any other" do
        assert render(Watchlist.ErrorView, "505.json", []) ==
               %{errors: %{detail: "Server internal error"}}
      end
    end

    When we run the test again, everything should be working properly.

    Conclusion

    When writing code, we need to have confidence in what we do, and TDD is a powerful tool for this. The feedback it gives us can make things clearer when we’re not sure how to implement a feature. The faster this feedback loop is, the faster we’ll move forward, even with new technologies.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by:
    Jader Correa is Brazilian, member of Ruby and Elixir communities and passionate about teaching, helping others to start writing code and playing the guitar.