My thoughts on testing Phoenix Controller Plug setup

Phoenix comes with a few handy ways to test its components. For example, it has Phoenix.ControllerTest module, which helps us make a request that goes through the application endpoint all the way up to the controller. Phoenix also generates a ConnCase module along with the project to better use Phoenix.ControllerTest during testing, with a dedicated setup block.

There are a few places, however, where we could further streamline testing in a Phoenix application. One such place is testing how a Controller is set up to use a Plug.

For example, consider this controller:

defmodule ExampleWeb.PageController do
  use ExampleWeb, :controller

  plug RequireClaims,
       index: "page:read",
       show: "page:read",
       create: "page:write",
       delete: "page:write"

  def index(conn, params) do
    # some code
  end

  def show(conn, params) do
    # some code
  end

  def create(conn, params) do
    # some code
  end

  def delete(conn, params) do
    # some code
  end
end

This is a pretty straightforward Phoenix controller which uses a custom plug, RequireClaims. As you might guess, RequireClaims does authorization based on claims and checks whether the claims assigned to the connection will allow it to invoke a controller action.

In the above example, we can see that index and show actions need "page:read" claim, whereas create and delete actions need "page:write" claim. Now let us see how we can go about testing it.

Request Testing the Plug (the conventional approach)

The regular Phoenix way to test this would be to send a request to this controller with different sets of claims and test the behavior of route and the connection.

The test file will look somewhat like this:

defmodule ExampleWeb.PageControllerTest do
  use ExampleWeb.ConnCase

  describe "index/2 (GET /page)" do
    # <--- Relevant code starts here
    # Testing without proper claims
    test "responds with 403 when proper claims aren't set", %{conn: conn} do
      conn = put_claims(conn, ["page:write"])

      conn = get(conn, "/page")

      assert conn.status == 409
      assert conn.halt
    end

    # <--- Relevant code starts here
    # Testing with proper claims
    test "works when claims are properly set", %{conn: conn} do
      conn = put_claims(conn, ["page:read"])

      conn = get(conn, "/page")

      refute conn.status == 409
      refute conn.halt

      # more assertions
    end
  end

  # .... more code

  def put_claims(conn, claims) do
    # some code that sets the claims for the conn
  end
end

This works well to test whether the controller is using the plug correctly. However, one might say that we’re testing the behavior of the plug itself instead of testing just the controller’s usage of the plug. But I think that is fine, since we’re only doing it for one controller.

The problem comes when we start using this plug for many controllers. If we add these tests for each controller (and its actions) it will be testing the behavior of this plug over and over again, slowing down our test suite (since request tests are relatively expensive). Moreover, we’re coupling the controller tests (which should indicate the controller’s behavior) to the plug’s behavior. So, if we were to change the behavior of the plug, we will have to update these tests in all the controllers even if the plug calls in the controllers stay the same.

I think you can probably see where I am going with this..

We don’t need to write a traditional controller test to test whether this plug is correctly being used in the controller file. As long as we have tested the plug itself (hopefully in its own test file with all the uses cases), all we need is just to test how it is being used in the controller.

Testing the Controller’s Plug usage (The goal)

Based on what we concluded in the previous section, our goal is to be able to do something like this:

defmodule ExampleWeb.PageControllerTest do
  use ExampleWeb.ConnCase
  import ExampleWeb.TestHelpers.ControllerPlugUsage # <-- line added

  # <------ New describe block added
  describe "plug RequireClaims" do
    test "sets up `RequireClaims` plug with correct options" do
      # uses_plug/2 comes from `ControllerPlugUsage` module
      assert uses_plug?(
        ExampleWeb.PageController,
        plug: RequireClaims,
        options: [
          index: "page:read",
          show: "page:read",
          create: "page:write",
          delete: "page:write"
        ]
      )
    end
  end
  # <---- New describe block end

  describe "index/2 (GET /page)" do
    # <------ Tests removed
    # Remove plug specific tests and just keep the happy path to test the
    # controller action's behavior

    test "works when claims are properly set", %{conn: conn} do
      conn = put_claims(conn, ["page:read"])

      conn = get(conn, "/page")

      refute conn.status == 409
      refute conn.halt

      # more assertions
    end
  end

  # .... more code

  def put_claims(conn, claims) do
    # some code that sets the claims for the conn
  end
end

The above file just tests whether the plug RequireClaims is configured correctly in the controller, instead of testing the behavior of the plug again. As a result of this, our tests will run much faster and are easier to follow.

Now let’s work on getting the code above to work.

Testing the Controller’s Plug usage (Making it work)

Now that we have seen the intended usage, it’s time to write the module which defines the uses_plug? function. However, before doing that we need a way to inspect a controller’s list of plugs.

Upon looking at phoenix’s Controller Pipeline module , it appears that every time the plug macro is called, the module attribute @plug accumulates new plug with its parameters. Also, since that module attribute is never reset, we can conclude towards the end of the compilation of any phoenix controller all its plugs are present in that module attribute.

(Optional) Breaking the black-box: Controller @plug attribute

For those interested to dig a little deeper, here’s how @plugs in the controller would look like:

[
  {RequireClaims, [index: "page:read", show: "page:read"...], true},
  {:put_new_view, TestView, true},
  {:put_new_layout, {ExampleWeb.LayoutView, :app}, true}
]

As you can see a plug in the list of @plugs is a three element tuple, where the first element is either a plug module or a function, the second element is a list of options/parameters that the plug takes, and the third element appears to be a boolean. Actually, the third element is actually used for storing guard clauses for invoking a plug. At first, we will be ignoring the third element and set it to true in order to make the code easier to follow, but I promise I’ll get to it at the end!

Adding __plugs__/0 controller function for plug introspection

Now we know that @plugs attribute of a controller has information about its plugs and their configuration. So, a simple way to get all the plugs would be to define a function in the controller which returns the module attribute @plugs.

Here’s how it will look:

defmodule ExampleWeb.PageController do
  use ExampleWeb, :controller

  plug RequireClaims,
       index: "page:read",
       show: "page:read",
       create: "page:write",
       delete: "page:write"

  def index(conn, params) do
    # some code
  end

  def show(conn, params) do
    # some code
  end

  def create(conn, params) do
    # some code
  end

  def delete(conn, params) do
    # some code
  end

  # <--- New code added here
  # Define a function which returns the module attribute `@plugs`
  def __plugs__, do: @plugs
end

It is very important to define this function after all the plug calls because each plug call updates the @plugs attribute as the attribute is registered with accumulate set to true.

For more information on accumulate: true, visit this hexdocs page

This function would return the list of plugs used by the controller:

iex(1)> ExampleWeb.PageController.__plugs__()
[
  {RequireClaims, [index: "page:read", show: "page:read" ...], true},
  {:put_new_view, TestView, true},
  {:put_new_layout, {ExampleWeb.LayoutView, :app}, true}
]

Writing the test helper uses_plug?/2

With __plugs__/0 function all set for the controller, we can go ahead create the test helper uses_plug?/2 in a new module ExampleWeb.TestHelpers.ControllerPlugUsage:

defmodule ExampleWeb.TestHelpers.ControllerPlugUsage do
  @moduledoc """
  Defines a list of functions that can be used to test a controller's plug
  usage without having to use request tests to test the plug's behavior
  again.
  """

  @doc """
  Returns `true` if the given plug is used by the given controller with the
  given opts (options)

  NOTE: Assumes there are no guard clauses in the plug call
  """
  def uses_plug?(controller, plug: plug, opts: opts) do
    expected_plug = {
      plug, opts, true
    }

    expected_plug in controller.__plugs__()
  end
end

Now, we can use this module to test whether a plug is being used by a controller in the way we would expect it to:

iex> ExampleWeb.TestHelpers.ControllerPlugUsage.uses_plug?(
...>   ExampleWeb.PageController,
...>   plug: RequireClaims,
...>   opts: [
...>    index: "page:read",
...>    show: "page:read",
...>    create: "page:write",
...>    delete: "page:write",
...>   ]
...> )
true

Taking it a step further (some metaprogramming knowledge would help here)

Let’s take it a step further. Instead of having to define __plugs__/0 function at the end of every controller, it would be nice to programatically insert that function definition at the end of a controller’s definition. The perfect place to do something like that is the lib/example_web.ex file, which defines quoted blocks that inject behavior into a controller. So, let’s update ExampleWeb.controller/0 function to accomplish that:

defmodule ExampleWeb do
  # ... some code

  def controller do
    quote do
      use Phoenix.Controller, namespace: ExampleWeb

      import Plug.Conn
      alias ExampleWeb.Router.Helpers, as: Routes

      # Add this line
      @before_compile ExampleWeb.ControllerBeforeCompile
    end
  end

  # Add this module which adds a before_compile hook which is responsible
  # for defining `__plugs__/0` function used for testing
  defmodule ControllerBeforeCompile do
    @moduledoc false

    defmacro __before_compile__(_macro_env) do
      quote do
        def __plugs__, do: @plugs
      end
    end
  end

  # .. some more code
end

The above @before_compile hook defines a function at the end of a controller. We needed to use @before_compile here instead of just defining it in the controller function’s quoted block, because all the plug calls happen after the quoted expression is evaluated. This means we need to wait for the entire body of the controller to be evaluated before defining the function, otherwise @plugs attribute won’t be up-to-date. Furthermore, since we can only use module attributes during the module’s compilation, we had to use @before_compile instead of @after_compile.

For more information on @before_compile and other hooks check out this hexdocs page

Now, whenever we define a controller using use ExampleWeb, :controller, it will also define a function __plugs__/0 which returns the list of plugs a controller is configured to work with along with their parameters/options.

iex(1)> defmodule TestController do
...(1)>   use ExampleWeb, :controller
...(1)>   plug :plug1, option1: :value1
...(1)>   plug :plug2, option2: :value2
...(1)>   def plug1(conn, _), do: conn
...(1)>   def plug2(conn, _), do: conn
...(1)> end
{:module, TestController,
 <<70, 79, 82, 49, 0, 0, 23, 248, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 2, 80,
   0, 0, 0, 51, 21, 69, 108, 105, 120, 105, 114, 46, 84, 101, 115, 116, 67, 111,
   110, 116, 114, 111, 108, 108, 101, 114, 8, ...>>, {:plug2, 2}}
iex(2)> TestController.__plugs__()
[
  {:plug1, [option1: :value1], true},
  {:plug2, [option2: :value2], true},
  {:put_new_view, TestView, true},
  {:put_new_layout, {ExampleWeb.LayoutView, :app}, true}
]

Conclusion

Controller tests (request tests) allow us to test router + controller + plugs as a black box. But, in order to speed up plug-specific controller tests and avoid testing plug behaviors multiple times, we can just test the plug calls in a phoenix controller. This post shows an example of how we can leverage some tools that elixir/phoenix provides to accomplish that.

I have similar posts coming up as I have been thinking of streamlining testing in the Elixir ecosystem for a few years now. Next up will be: Testing Phoenix Router Pipelines and Routes.

(Optional) Making it work for guard clauses (a little bit more meta)

I promised I’ll get to it, so here it is..

To make the uses_plug?/2 helper work with guards we will have to understand how a Phoenix.Controller stores a guard corresponding to a plug.

Let’s take a look at a controller that has plugs with guards, and inspect it:

iex(1)> defmodule TestController do
...(1)>   use ExampleWeb, :controller
...(1)>   plug :plug1, [option1: :value1] when action in [:index]
...(1)>   plug :plug2, [option2: :value2] when action in [:show]
...(1)>   def plug1(conn, _), do: conn
...(1)>   def plug2(conn, _), do: conn
...(1)> end
{:module, TestController,
  <<70, 79, 82, 49, 0, 0, 24, 216, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 2, 91,
  0, 0, 0, 53, 21, 69, 108, 105, 120, 105, 114, 46, 84, 101, 115, 116, 67, 111,
  110, 116, 114, 111, 108, 108, 101, 114, 8, ...>>, {:plug1, 2}}
iex(2)> TestController.__plugs__()
[
  {:plug2, [option2: :value2],
    {:in, [line: 4], [{:action, [line: 4], nil}, [:show]]}},
  {:plug1, [option1: :value1],
    {:in, [line: 3], [{:action, [line: 3], nil}, [:index]]}},
  {:put_new_view, TestView, true},
  {:put_new_layout, {ExampleWeb.LayoutView, :app}, true}
]

As you can see above, the guards are expressed as quoted expressions. So, my first thought was that we can send a quoted guard clause as part of the keyword list options in uses_plug?/2. So the function will look somewhat like this:

defmodule ExampleWeb.TestHelpers.ControllerPlugUsage do
  # ...

  @doc """
  Returns `true` if the given plug is used by the given controller with the
  given opts (options)

  If no `guard` is given, it defaults to `true` (Just like the controller)
  """
  def uses_plug?(controller, plug: plug, opts: opts, guard: guard) do
    expected_plug = {
      plug, opts, guard
    }

    expected_plug in controller.__plugs__()
  end

  def uses_plug?(controller, plug: plug, opts: opts) do
    uses_plug?(controller, plug: plug, opts: opts, guard: true)
  end
end

You might think this will work, but it gets complicated to evaluate quoted expressions especially when the contexts are different.

What that means is, this will return false:

iex(1)> ExampleWeb.TestHelpers.ControllerPlugUsage.uses_plug?(
...(1)>   TestController,
...(1)>   plug: :plug1,
...(1)>   opts: [option1: :value1],
...(1)>   guard: quote do: action in [:index]
...(1)> )
false

Why is that? It’s because the contextual meta-data (the second element of the quoted expression) of the plug’s guard will be different when it’s quoted in the controller vs when it’s quoted somewhere else (test file or iex shell).

I tried many ways to solve this and the simplest way, in my opinion, is to check if the raw value of the quoted expressions are the same using Macro.to_string/1 function.

iex(1)> Macro.to_string(quote do: action in [:index])
"action in [:index]"

As you can see above, this is independent of the context in which the expression was quoted. So, let us change our helper to check for equality of stringified quoted expressions instead of regular quoted expressions:

defmodule ExampleWeb.TestHelpers.ControllerPlugUsage do
  # ...

  @doc """
  Returns `true` if the given plug is used by the given controller with the
  given opts (options) and given `guard`.

  If no `guard` is given, it defaults to `true` (Just like the controller)

  # <--- Important note here
  NOTE: This checks if the raw string of the guards are as expected and doesn't
  take into account the ambiguity of the expression.

  For example, these two even though are programatically equivalent, they won't
  be equal in this case because their string forms are not equal:

  `not action in [:create] and not :something in conn.private`
  and
  `not (action in [:create] or :something in conn.private)`
  """
  def uses_plug?(controller, plug: plug, opts: opts, guard: guard) do
    expected_plug = plug_with_stringified_guard({plug, opts, guard})

    plugs_with_stringified_guard =
      Enum.map(controller.__plugs__(), &plug_with_stringified_guard/1)

    expected_plug in plugs_with_stringified_guard
  end

  def uses_plug?(controller, plug: plug, opts: opts) do
    uses_plug?(controller, plug: plug, opts: opts, guard: true)
  end

  # Convert to string if `guard` is `quoted`
  # Do not convert to string if `guard` is `true` or unquoted
  defp plug_with_stringified_guard({plug, opts, guard}) do
    if Macro.quoted_literal?(guard) do
      {plug, opts, Macro.to_string(guard)}
    else
      {plug, opts, guard}
    end
  end
end

In the above module, we now convert both the quoted guard clause from the controller and the quoted guard clause from the test file to raw strings. This modification allows us to use uses_plug?/2 with guard clauses:

iex(1)> ExampleWeb.TestHelpers.ControllerPlugUsage.uses_plug?(
...(1)>   TestController,
...(1)>   plug: :plug1,
...(1)>   opts: [option1: :value1],
...(1)>   guard: quote do: action in [:index]
...(1)> )
true

As the function doc for uses_plug?/2 implies, there is a flaw in this way of testing the equality of guards. We’re not taking their logic into consideration but only their raw string form. This means this plug tests only the literal equivalence of guards, and not the logical equivalence.

I’m sure there’s a better way to test guards equivalence using a combination of Macro.expand/2 function and the fact that we only have access to conn, action and controller variables in a plug’s guard clause. Let us leave that for a future blog post… 👋