How do I draw with inverted color in graphicsmagick?

I use graphicsmagick to add text to photos; but I'm having problems selecting the proper color for my text.

With graphicsmagick I can use a command such as this

convert -fill yellow -draw "... text ..." ...

to write yellow text on an image.

Is there a way to write with color inversion instead? What I mean is: Where the underlying picture is black, I want the text to be white; where the underlying picture is white, I want the text to be black; where the underlying picture is a certain color, I want the text to be in its complement color.

EDIT

Please note that "No, it can't be done" is a perfectly good answer to this question if what I want is impossible.

You can use -compose Difference to overlay your desired text:

Original:

Then create a text gif.:

convert -background transparent -fill yellow label:Rose label_white.gif

Then compose them together using difference:

composite -compose Difference -gravity North label_white.gif rose: compose_diff.gif

You can find more compositing info here.