Why do we need flatMap (in general)?

I have been looking into FP languages (off and on) for some time and have played with Scala, Haskell, F#, and some others. I like what I see and understand some of the fundamental concepts of FP (with absolutely no background in Category Theory - so don't talk Math, please).

So, given a type M[A] we have map which takes a function A=>B and returns a M[B]. But we also have flatMap which takes a function A=>M[B] and returns a M[B]. We also have flatten which takes a M[M[A]] and returns a M[A].

In addition, many of the sources I have read describe flatMap as map followed by flatten.

So, given that flatMap seems to be equivalent to flatten compose map, what is its purpose? Please don't say it is to support 'for comprehensions' as this question really isn't Scala-specific. And I am less concerned with the syntactic sugar than I am in the concept behind it. The same question arises with Haskell's bind operator (>>=). I believe they both are related to some Category Theory concept but I don't speak that language.

I have watched Brian Beckman's great video Don't Fear the Monad more than once and I think I see that flatMap is the monadic composition operator but I have never really seen it used the way he describes this operator. Does it perform this function? If so, how do I map that concept to flatMap?

BTW, I had a long writeup on this question with lots of listings showing experiments I ran trying to get to the bottom of the meaning of flatMap and then ran into this question which answered some of my questions. Sometimes I hate Scala implicits. They can really muddy the waters. :)

FlatMap, known as "bind" in some other languages, is as you said yourself for function composition.

Imagine for a moment that you have some functions like these:

def foo(x: Int): Option[Int] = Some(x + 2)
def bar(x: Int): Option[Int] = Some(x * 3)

The functions work great, calling foo(3) returns Some(5), and calling bar(3) returns Some(9), and we're all happy.

But now you've run into the situation that requires you to do the operation more than once.

foo(3).map(x => foo(x)) // or just foo(3).map(foo) for short

Job done, right?

Except not really. The output of the expression above is Some(Some(7)), not Some(7), and if you now want to chain another map on the end you can't because foo and bar take an Int, and not an Option[Int].

Enter flatMap

foo(3).flatMap(foo)

Will return Some(7), and

foo(3).flatMap(foo).flatMap(bar)

Returns Some(15).

This is great! Using flatMap lets you chain functions of the shape A => M[B] to oblivion (in the previous example A and B are Int, and M is Option).

More technically speaking; flatMap and bind have the signature M[A] => (A => M[B]) => M[B], meaning they take a "wrapped" value, such as Some(3), Right('foo), or List(1,2,3) and shove it through a function that would normally take an unwrapped value, such as the aforementioned foo and bar. It does this by first "unwrapping" the value, and then passing it through the function.

I've seen the box analogy being used for this, so observe my expertly drawn MSPaint illustration:

This unwrapping and re-wrapping behavior means that if I were to introduce a third function that doesn't return an Option[Int] and tried to flatMap it to the sequence, it wouldn't work because flatMap expects you to return a monad (in this case an Option)

def baz(x: Int): String = x + " is a number"

foo(3).flatMap(foo).flatMap(bar).flatMap(baz) // <<< ERROR

To get around this, if your function doesn't return a monad, you'd just have to use the regular map function

foo(3).flatMap(foo).flatMap(bar).map(baz)

Which would then return Some("15 is a number")