Entity Framework: ObjectSet and its (generics) variance

I use: EntityFramework + POCO

Here is the thing:

public interface IBaseType
{
    int Id { get; set; }
}

public class BaseType : IBaseType
{
    public virtual int Id { get; set; }
}

public class DerivedType : BaseType
{
}

The problem:

public class EntityFetcher<T> where T : BaseType
{
    public object GetById(int id)
    {
        ObjectSet<T> objectSet = (ObjectSet<T>)GetTheObjectSet(typeof(T)); 

        return objectSet.SingleOrDefault((o) => o.Id == id);
    }
}

If T is BaseType this all works perfectly, but: The problem is that in EntityFramework when one class inherits another they share the ObjectSet and, therefore, if T is DerivedType then the GetTheObjectSet returns ObjectSet<BaseType>, which cannot be cast to ObjectSet<DerivedType>.

Is there a way to actually cast this this thing or somehow else execute the SingleOrDefault? Can those things be cast using the IObjectSet<> and IBaseType?

The answer to this casting problem was as follows:

public T GetById(int id)
{
    // The line that throws InvalidCast with T = DerivedType
    //ObjectSet<T> objectSet = (ObjectSet<T>)GetTheObjectSet(typeof(T));

    // This is a valid cast since ObjectSet<T> is derived from ObjectQuery
    ObjectQuery objectQuery = (ObjectQuery)GetTheObjectSet(typeof(T));
    return objectQuery.OfType<T>().SingleOrDefault((e) => e.Id == id);
}

The solution was to cast ObjectSet to ObjectQuery and do the query there. The most important part here is that ObjectQuery is not generic, so the cast goes through fine.

I must give some credits to Bennor McCarthy as he was the one to point me to OfType + casting to ObjectQuery (the fact that ObjectSet : ObjectQuery). Thanks!