Why not a Phantom class which extends Functor Contravariant?

I am playing around with the Data.Functor.Contravariant. The phantom method caught my eye:

phantom :: (Functor f, Contravariant f) => f a -> f b
phantom x = () <$ x $< ()

Or, more specifically, the annotation to it:

If f is both Functor and Contravariant then by the time you factor in the laws of each of those classes, it can't actually use it's argument in any meaningful capacity. This method is surprisingly useful. Where both instances exist and are lawful we have the following laws: fmap f ≡ phantom, contramap f ≡ phantom

Since fmap f ≡ contramap f ≡ phantom, why do we need Contravariant and Functor instances? Isn't it handier to do this thing the other way: create an instance for one class Phantom, which introduces the phantom method, and then automatically derive instances for Functor and Contravariant?

class Phantom f where
    phantom :: f a -> f b

instance Phantom f => Functor f where
    fmap _f = phantom

instance Phantom f => Contravariant f where
    contramap _f = phantom

We will rid the programmer of the necessity to rewrite this phantom twice (to implement fmap and contramap, which are const phantom, as stated in the annotation) when implementing instances for Contravariant and Functor. We will allow writing one instance instead of two! Besides, it seems nice and idiomatic to me to have classes for all 4 cases of variance: Functor, Contravariant, Invariant (yet, some suggest using Profunctor interface instead of Invariant), and Phantom.

Also, isn't it a more efficient approach? () <$ x $< () requires two traverses (as much as we can traverse a phantom functor...), as long as the programmer might carry this transformation out a bit faster. As far as I understand, the current phantom method can't be overridden.

So, why didn't the library developers choose this way? What are the pros and cons of the current design and the design I spoke of?

To avoid the overlapping instances mentioned by amalloy you could define a newtype which can be used with DerivingVia:

{-# LANGUAGE DerivingVia #-}

import Data.Functor.Contravariant hiding (phantom)

class (Functor f, Contravariant f) => Phantom f where
  phantom :: f a -> f b

newtype WrappedPhantom f a = WrappedPhantom (f a)

instance Phantom f => Phantom (WrappedPhantom f) where
  phantom (WrappedPhantom x) = WrappedPhantom (phantom x)

instance Phantom f => Functor (WrappedPhantom f) where
  fmap _ = phantom

instance Phantom f => Contravariant (WrappedPhantom f) where
  contramap _ = phantom

-- example of usage:

data SomePhantom a = SomePhantom
  deriving (Functor, Contravariant) via WrappedPhantom SomePhantom

instance Phantom SomePhantom where
  phantom SomePhantom = SomePhantom

It's not quite as convenient as having the instances automatically, but it still means that you don't have to implement Functor and Contravariant instances manually.