Why does Finatra use flatMap and not just map?

This might be a really dumb question but I am trying to understand the logic behind using #flatMap and not just #map in this method definition in Finatra's HttpClient definition:

def executeJson[T: Manifest](request: Request, expectedStatus: Status = Status.Ok): Future[T] = {
  execute(request) flatMap { httpResponse =>
    if (httpResponse.status != expectedStatus) {
      Future.exception(new HttpClientException(httpResponse.status, httpResponse.contentString))
    } else {
      Future(parseMessageBody[T](httpResponse, mapper.reader[T]))
        .transformException { e =>
          new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
        }
    }
  }
}

Why create a new Future when I can just use #map and instead have something like:

execute(request) map { httpResponse =>
  if (httpResponse.status != expectedStatus) {
    throw new HttpClientException(httpResponse.status, httpResponse.contentString)
  } else {
    try {
      FinatraObjectMapper.parseResponseBody[T](httpResponse, mapper.reader[T])
    } catch {
      case e => throw new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
    }
  }
}

Would this be purely a stylistic difference and using Future.exception is just better style in this case, whereas throwing almost looks like a side-effect (in reality it's not, as it doesn't exit the context of a Future) or is there something more behind it, such as order of execution and such?

Tl;dr: What's the difference between throwing within a Future vs returning a Future.exception?

From a theoretical point of view, if we take away the exceptions part (they cannot be reasoned about using category theory anyway), then those two operations are completely identical as long as your construct of choice (in your case Twitter Future) forms a valid monad.

I don't want to go into length over these concepts, so I'm just going to present the laws directly (using Scala Future):

import scala.concurrent.ExecutionContext.Implicits.global

// Functor identity law
Future(42).map(x => x) == Future(42)

// Monad left-identity law
val f = (x: Int) => Future(x)
Future(42).flatMap(f) == f(42) 

// combining those two, since every Monad is also a Functor, we get:
Future(42).map(x => x) == Future(42).flatMap(x => Future(x))

// and if we now generalise identity into any function:
Future(42).map(x => x + 20) == Future(42).flatMap(x => Future(x + 20))

So yes, as you already hinted, those two approaches are identical.

However, there are three comments that I have on this, given that we are including exceptions into the mix:

1. Be careful - when it comes to throwing exceptions, Scala Future (probably Twitter too) violates the left-identity law on purpose, in order to trade it off for some extra safety.

Example:

import scala.concurrent.ExecutionContext.Implicits.global

def sneakyFuture = {
  throw new Exception("boom!")
  Future(42)
}

val f1 = Future(42).flatMap(_ => sneakyFuture)
// Future(Failure(java.lang.Exception: boom!))

val f2 = sneakyFuture
// Exception in thread "main" java.lang.Exception: boom!

2. As @randbw mentioned, throwing exceptions is not idiomatic to FP and it violates principles such as purity of functions and referential transparency of values.

Scala and Twitter Future make it easy for you to just throw an exception - as long as it happens in a Future context, exception will not bubble up, but instead cause that Future to fail. However, that doesn't mean that literally throwing them around in your code should be permitted, because it ruins the structure of your programs (similarly to how GOTO statements do it, or break statements in loops, etc.).

Preferred practice is to always evaluate every code path into a value instead of throwing bombs around, which is why it's better to flatMap into a (failed) Future than to map into some code that throws a bomb.

3. Keep in mind referential transparency.

If you use map instead of flatMap and someone takes the code from the map and extracts it out into a function, then you're safer if this function returns a Future, otherwise someone might run it outside of Future context.

Example:

import scala.concurrent.ExecutionContext.Implicits.global

Future(42).map(x => {
  // this should be done inside a Future
  x + 1
})

This is fine. But after completely valid refactoring (which utilizes the rule of referential transparency), your codfe becomes this:

def f(x: Int) =  {
  // this should be done inside a Future
  x + 1
}
Future(42).map(x => f(x))

And you will run into problems if someone calls f directly. It's much safer to wrap the code into a Future and flatMap on it.

Of course, you could argue that even when using flatMap someone could rip out the f from .flatMap(x => Future(f(x)), but it's not that likely. On the other hand, simply extracting the response processing logic into a separate function fits perfectly with the functional programming's idea of composing small functions into bigger ones, and it's likely to happen.