RabbitMQ RPC across multiple rabbitMQ instances

I have three clients each with their own RabbitMQ instances and I have an application (let's call it appA) that has its own RabbitMQ instance, the three client applications (app1, app2, app3) wants to make use of a service on appA.

The service on appA requires RPC communication, app1, app2 and app3 each has a booking.request queue and a booking.response queue.

With the shovel plugin, I can forward all booking.request messages from app1-3 to appA:

Shovel1 
virtualHost=appA, 
name=booking-request-shovel, 
sourceURI=amqp://userForApp1:password@app1-server/vhostForApp1
queue=booking.request
destinationURI=amqp://userForAppA:password@appA-server/vhostForAppA
queue=booking.request

setup another shovel to get booking requests from app2 and app3 to appA in the same way as above.

Now appA will respond to the request on the booking.response queue, I need the booking response message on rabbitMQ-appA to go back to the correct booking.response queue either on app1, app2 or app3, but not to all of them - how do I setup a shovel / federated queue on rabbitMQ-appA that will forward the response back to the correct rabbitMQ (app1, app2, app3) that is expecting a response in their own booking.response queue?

All these apps are using spring-amqp (in case that's relevant) Alternatively, I could setup a rabbitMQ template in Spring that listens to multiple rabbitMQ queues and consumes from each of them.

From the docs, this what a typical consumer looks like:

<rabbit:listener-container connection-factory="rabbitConnectionFactory">
    <rabbit:listener queues="some.queue" ref="somePojo" method="handle"/>
</rabbit:listener-container>

Is it possible to specify multiple connection factories in order to do this even if the connection factories are to the same instance of RabbitMQ, but just different vhosts:

Update:

Based on Josh's answer, I'd have multiple connection factories:

 <rabbit:connection-factory
                id="connectionFactory1"
                port="${rabbit.port1}"
                virtual-host="${rabbit.virtual1}"
                host="${rabbit.host1}"
                username="${rabbit.username1}"
                password="${rabbit.password1}"
                connection-factory="nativeConnectionFactory" />

 <rabbit:connection-factory
                id="connectionFactory2"
                port="${rabbit.port2}"
                virtual-host="${rabbit.virtual2}"
                host="${rabbit.host2}"
                username="${rabbit.username2}"
                password="${rabbit.password2}"
                connection-factory="nativeConnectionFactory" />

Then I would use the SimpleRoutingConnectionFactory to wrap both connection-factories:

<bean id="connectionFactory" class="org.springframework.amqp.rabbit.connection.SimpleRoutingConnectionFactory">
    <property name="targetConnectionFactories">
        <map>
            <entry key="#{connectionFactory1.virtualHost}" ref="connectionFactory1"/>
            <entry key="#{connectionFactory2.virtualHost}" ref="connectionFactory2"/>
        </map>
    </property>
</bean>

Now when I declare my rabbitMQ template, I would point it to the SimpleRoutingConnectionFactory instead of the individual connection factories:

<rabbit:template id="template" connection-factory="connectionFactory" />

... and then use the template as I would normally use it ...

<rabbit:listener-container
        connection-factory="connectionFactory"
        channel-transacted="true"
        requeue-rejected="true"
        concurrency="${rabbit.consumers}">
        <rabbit:listener queues="${queue.booking}" ref="TransactionMessageListener" method="handle"  />
</rabbit:listener-container>

// and messages are consumed from both rabbitMQ instances

... and ...

  @Autowired
  private AmqpTemplate template;

  template.send(getExchange(), getQueue(), new Message(gson.toJson(message).getBytes(), properties));

// and message publishes to both queues

Am I correct?

Take a look at org.springframework.amqp.rabbit.connection.AbstractRoutingConnectionFactory. It will allow you to create multiple connection factories to different vhosts or different rabbitmq instances. We are using it for a multi tenant rabbitmq application.