Getting a refined type by matching a member value corresponding to a type member

I have a container for heterogeneous types

trait Elem

trait Obj {
  type E <: Elem
  def elem: E
}

trait Foo extends Elem

There should not be any sub-classes of Obj. I am looking for suggestions as how to elegantly match an instance of Obj against a number of element types to gain a refined Obj.

For example:

def withFooObj(obj: Obj { type E = Foo }) = ()

def test(obj: Obj): Unit = obj.elem match {
  case f: Foo =>
    withFooObj(obj.asInstanceOf[Obj { type E = Foo }]) // !
  case _ =>
}

What would be an elegant idea to get rid of the cast? Ideally it would work as a pattern match.

---

For example, I can create individual extractors:

object IsFoo {
  def unapply(obj: Obj): Option[Obj { type E = Foo }] =
    if (obj.elem.isInstanceOf[Foo]) Some(obj.asInstanceOf[Obj { type E = Foo }])
    else None
}

def test(obj: Obj): Unit = obj match {
  case IsFoo(x) => withFooObj(x)
  case _ =>
}

But I have many sub-types of Elem and would prefer to have a generic extractor.

---

Edit: Since this seems to be a tough one, I want to add another relaxation: It is allowed to change Obj to contain a type parameter instead of a type member (if it helps). It is not allowed to require sub-classes of Obj.

I would expect this to be impossible. Consider (assuming class Foo extends Elem)

val obj1 = new Obj {
  type E = Elem
  def elem = new Foo
}

Since obj.elem does match f: Foo, but obj1 doesn't actually have type Obj { type E = Foo }, you do have to cast. This cast is actually safe in this case, but only because Obj doesn't have e.g. any methods taking E. Actually, it isn't safe, since the value of the first call to elem having class Foo doesn't mean all calls to elem will return Foo.

EDIT: if you are all right with merely hiding the cast in the extractor, you may be able to do something like this:

case class IsElem[T <: Elem]()(implicit ct: ClassTag[T]) {
  def unapply(obj: Obj): Option[Obj { type E = T }] =
    if (ct.runtimeClass.isInstance(obj.elem)) Some(obj.asInstanceOf[Obj { type E = T }])
      else None
}

val IsFoo = IsElem[Foo]
val IsBar = IsElem[Bar]
...

EDIT2: with type parameters, you can do

(obj, obj.elem) match {
  case (obj: Obj[Foo] @unchecked, f: Foo) => ...
  case (obj: Obj[Bar] @unchecked, f: Bar) => ...
}

Obviously, this doesn't help safety issues.