With the memento design pattern, is it wrong to have the Caretaker as an aggregate of the Originator?
Not necessarily. The main purpose behind Memento is that it is possible to deal with different Originators in uniform, transparent way (get Memento, pass it back later to restore state).
The aggregate relation doesn't necessarily prevent that. Different instances of Caretaker might be aggregates of different Originators (although one instance of Caretaker deals with only one instance of Originator) and still get the benefits of transparent state restoration.