Association for one model belonging to two different models

Which type of rails association should I use for this scenario?

A Question has many Answers, and a User also has many Answers.

A polymorphic association seems very close, but I need the Answer to the Question and the Answer that the User gives to point to the same object. It seems different from example on the association guide because in that example the Picture object is different for an Employee and a Product. Using their example, I need an Employee's picture and a Product's picture to be the same Picture.

I think I need a structure like

-----------------------------------------------
answer_id | question_id | user_id | answer_text
-----------------------------------------------
|    1    |      5      |    10   |   stuff   |
|    2    |      3      |    6    |   stuff 2 |
|    3    |      2      |    10   |   stuff 3 |

I could make answers belong to questions, and add the user_id manually, but that doesn't feel very "railsy."

Any thoughts appreciated.

Wouldn't this work for you?

class User < ActiveRecord::Base
 has_many :answers
end

class Question < ActiveRecord::Base
 has_many :answers
end

class Answer < ActiveRecord::Base
 belongs_to :question
 belongs_to :user
end

You table structure will be like

answers
   -question_id
   -user_id
   -answer_text