Should I almost always use virtual inheritance?

I see from Virtual inheritance doesn't break static composition? that virtual inheritance adds sizeof(pointer) to an object's memory footprint. Other than that, are there any drawbacks to me just using virtual inheritance by default, and conventional inheritance only when needed? It seems like it'd lead to more future-proof class design, but maybe I'm missing some pitfall.

The drawbacks are that

1. All classes will have to initialize all its virtual bases all the time (e.g. if A is virtual base of B, and C derives from B, it also have to initialize A itself).
2. You have to use more expensive dynamic_cast everywhere you use a static_cast (may or may not be the issue, depending on your system and whether your design requires it).

Point 1 alone makes it not worth it, since you can't hide your virtual bases. There is almost always a better way.