How does akka-http serializers/deserializers work?

2020-11-10

scala akka-http marshaller

Well, well, it’s time for me to try and explain how akka-http serializing / deserializing of HTTP payloads, or as they call it: marshalling/unmarshalling.

I used mdoc to write this so the result you’ll see is compiled and should work. The command I’ve used is:

mdoc -w -i serializers.md  --classpath (cs fetch --classpath com.typesafe.akka::akka-stream:2.6.8 com.typesafe.akka::akka-http:10.2.1)

This uses cs (coursier) to build a classpath with the required dependencies. It’s passed to mdoc that watches my serializers.md file and output the compiled result in out/serializers.md. By default, in watch mode, mdoc also host a webserver with a rendered HTML version at http://localhost:4000/serializers.md.

Before we get into it, we’ll set a few imports to make sure everything is available for the rest of this post.

import akka.http.scaladsl.model._
import akka.http.scaladsl.model.StatusCodes._
import akka.http.scaladsl.server._
import akka.http.scaladsl.server.Directives._

import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global

One suggestion I can do for the readers of this post is this: as you follow along, copy the snippet into a Scala worksheet in your favourite editor, and play with the code. You’ll get a much better understanding. (The easiest approach is probably to create your worksheet in a project where you know those dependencies will be available and thus you get IntelliSense from your IDE).

I’ll do it, and I’ll drop my worksheet at the end of the post if you want to grab it. Also, here’s the scaladoc for akka-http: https://doc.akka.io/api/akka-http/10.2.1/akka/http/scaladsl/index.html.

With these imports, we can define a simple route, like something you’ve most likely seen before. Either in our code, or in examples in the wild:

val helloRoute: Route =
  path("hello") {
    get {
      complete(OK, "Hello world")
    }
  }
// helloRoute: RequestContext => Future[RouteResult] = akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1480/551377008@353b0719

So what is going on, here, really. Because from a reading perspective, it’s quite simple: if the path is /hello returns a 200 OK with "Hello world" as the payload. But from a writing perspective, how does that work?

The first intuition is that complete is implemented somewhat like that: complete(status: StatusCode, body: String). Well, it’s not the case. Go ahead, follow the implementation using your IDE. Instead, you end up in RoutesDirectives and the implementation is:

def complete(m: => ToResponseMarshallable): StandardRoute =
    StandardRoute(_.complete(m))

So our first intuition is wrong. It’s disappointing, but the implementation can lead us into the real explanation. What’s ToResponseMarshallable. Well, again, follow the types using your editor. In this case, we’re taken to ToResponseMarshallable.scala. The full source is here. So, ToResponseMarshallable is a type class to turn a T into a HttpResponse. It does not help much, and if you look at some of the usage, they’re all related to the completion of a request/response cycle. Again, in this file, nothing related to a StatusCode or a String that would explain how this thing resolves. But we’ve got another lead: ToResponseMarshaller. It’s a type alias, and its implemented like this: type ToResponseMarshaller[T] = Marshaller[T, HttpResponse]. If you follow, the implementation, you can see other similar alias are defined:

package object marshalling {
  //#marshaller-aliases
  type ToEntityMarshaller[T] = Marshaller[T, MessageEntity]
  type ToByteStringMarshaller[T] = Marshaller[T, ByteString]
  type ToHeadersAndEntityMarshaller[T] = Marshaller[T, (immutable.Seq[HttpHeader], MessageEntity)]
  type ToResponseMarshaller[T] = Marshaller[T, HttpResponse]
  type ToRequestMarshaller[T] = Marshaller[T, HttpRequest]
  //#marshaller-aliases
}

At a first glance, this Marshaller thing seems like a converter. A way to turn A into B. And if we look at the code inside ToResponseMarshallable.scala, we can see it that if it can find a ToResponseMarshaller[A] then it can define a ToResponseMarshallable and we can call complete with it as a paramter. Don’t worry, we all agree that Marshaller, ToResponseMarshaller[A] and ToResponseMarshallable are super confusing.

Let’s define a class Toto and a ToResponseMarshaller[Toto] and see if we can complete(toto) in a route:

import akka.http.scaladsl.marshalling._

final case class Toto(message: String)
object Toto {
  //remember, ToResponseMarshaller[Toto] is a `Marshaller[T, HttpResponse]`
  implicit val trmToto: ToResponseMarshaller[Toto] =
    Marshaller.withFixedContentType(ContentTypes.`text/plain(UTF-8)`) { toto =>
      HttpResponse(status = OK, entity = HttpEntity(toto.message))
    }
}

The implementation here really, does not matter. Marshallers can be quite complex to implement. Here I looked at the source and tried to pick the easiest implementation possible. withFixedContentType is only one of the multiple functions you can use.

With that, let’s define a route:

val totoRoute: Route =
  path("toto") {
    get {
      complete(Toto("My message"))
    }
  }
// totoRoute: RequestContext => Future[RouteResult] = akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1480/551377008@1c92a549

It works! Note that I’ve removed the status code and I only supply a Toto which is implicitly turned into a ToResponseMarshallable because a ToResponseMarshaller[Toto] is available. At this point, it’s most likely still foggy for you. Don’t worry, we’ll keep digging.

The implementation relies on two tricks:

the magnet pattern
a type class hierarchy to turn T into a HttpResponse

The magnet pattern is complex and I won’t try to explain it here, instead you should read this later. But for the purpose of this post, we’ll assume the magnet pattern is a clever trick to allow multiple implementation of one method/function using one signature. This is why complete, as shown above, works for Toto, but also for (StatusCode, String).

The second one is more of a convenience rather than a requirement. akka-http support a hierarchy of Marshaller built on top of each other so that you don’t always have to Marshaller[A, Response] to build a response. Instead, if you have a A maybe you can just do a Marshaller[A, String] and let the hierarchy turn it into a that Marshaller[A, Response] via other predefined marshallers.

Can we do that with Tata? Most likely. Let’s try to avoid defining the full Marshaller and instead really on an intermediate type class in the hierarchy for us. If we define Tata as:

final case class Tata(message: String)

Note: I only defined Tata to avoid conflict with Toto during implicit type class resolution

Then, the only thing that matters is that message, a String. What’s available for us? I don’t know we’ll have to dig. The IDE and the Scaladoc are two good tools for that.

This outline one problem with type classes and the magnet pattern: discoverability. You’ll often find yourself looking through piles and piles of code to see how you could arrange your call to complete for it to a) combile and b) make sure it does what you want (headers, status, etc.).

object Tata {
  implicit val trmTata: ToEntityMarshaller[Tata] = Marshaller.StringMarshaller.compose[Tata](_.message)
}

In the example above, we built a ToEntityMarshaller[Tata] (it’s not a ToResponseMarshaller[Toto]), from the StringMarshaller. It saves us from dealing with a StatusCode because 200 OK is fine for us. It also deals with the content-type. But can we pass a Tata in a complete call. Can the ToEntityMarshaller be turned into a ToResponseMarshaller into a ToResponseMarshallable? Let’s try:

val tataRoute: Route =
  path("tata") {
    get {
      complete(Tata("My message"))
    }
  }
// tataRoute: RequestContext => Future[RouteResult] = akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1480/551377008@323e026d

Is the result equivalent? Unfortunately, you’ll have to try it to see (passing a HttpRequest into a Route is no easy task, this is not http4s). But it compiles so I assume that it works as intended (I know, it’s really naive, but whatever).

What does that mean for our original question: how can complete be called with a StatusCode and a String if the signature is: def complete(m: => ToResponseMarshallable): StandardRoute.

Let’s try to find something that turns a (StatusCode, String) into a ToResponseMarshallable. From our investigation, we know that ToResponseMarshaller and ToEntityMarshaller can be turned into ToResponseMarshallable implicitly.

Let’s use our IDE to try and find how this can happen:

Exploring with Intellij

We find this implicit function:

implicit def fromStatusCodeAndValue[S, T](implicit sConv: S => StatusCode, mt: ToEntityMarshaller[T]): TRM[(S, T)]

So if we have a S that is convertible to a StatusCode and a T that has an ToEntityMarshaller[T] instance, we can generate a ToResponseMarshaller for the pair (S, T) and thus a ToResponseMarshallable.

And by default, akka-http provides those:

in our case, S is a StatusCode so the sConv is implicitly available
T is String and we already saw (above) that there is a Marshaller.StringMarshaller available

One way to find this is by enabling expansion of implicit hints:

Hints demo

And stating the types explicitly, forcing the compiler to resolve implicits the way you want and telling you if something is missing:

Hints demo

There you have it. That’s how it’s possible to go from a function that takes a m: ToResponseMarshallable to a function where you can actually supply s: StatusCode, body: String.

Here is the Scala worksheet if you want to start from that:


import akka.http.scaladsl.model._
import akka.http.scaladsl.model.StatusCodes._
import akka.http.scaladsl.server._
import akka.http.scaladsl.server.Directives._

val helloRoute: Route =
  path("hello") {
    get {
      complete(OK, "Hello world")
    }
  }

import akka.http.scaladsl.marshalling._

final case class Toto(message: String)
object Toto {
  //remember, ToResponseMarshaller[Toto] is a `Marshaller[T, HttpResponse]`
  implicit val trmToto: ToResponseMarshaller[Toto] =
    Marshaller.withFixedContentType(ContentTypes.`text/plain(UTF-8)`) { toto =>
      HttpResponse(status = OK, entity = HttpEntity(toto.message))
    }
}

val totoRoute: Route =
  path("toto") {
    get {
      complete(Toto("My message"))
    }
  }

final case class Tata(message: String)
object Tata {
  implicit val trmTata: ToEntityMarshaller[Tata] = Marshaller.StringMarshaller.compose[Tata](_.message)
}

val tataRoute: Route =
  path("tata") {
    get {
      complete(Tata("My message"))
    }
  }

val helloRoute2: Route = {
  val myResponse: ToResponseMarshallable = (OK, "Hello world")
  // this is why is an `implicit sConv: S => StatusCode` in fromStatusCodeAndValue
  // so that it works with status codes and `Int`
  val myOtherResponse: ToResponseMarshallable = (200, "Hello world")
  path("hello") {
    get {
      complete(myResponse)
    } ~
      options {
        complete(myOtherResponse)
      }
  }
}