Destroying UpVotes on acts_as_votable

I have a rails app that is working fine using acts_as_votable. The like button upvotes the post count, and then switches to an un-like button and this down votes the post count.

My issue is, that since I started using the Public Activity gem, I can't find a way to remove likes from the feed. I have used the following loop in the activities index view:

<% @activities.each do |activity| %>
    <p>
        <% if activity.trackable %>
            <%= link_to activity.owner.name, activity.owner %>
            <%= render_activity activity %>
        <% end %>
    </p>
<% end %>

When I delete a comment, the entire line in the activity feed of 'FOO added a comment on BAR' disappears. However, because the acts as votable gem actually creates a downvote rather than destroying the upvote, the line 'FOO liked BAR' still appears and would be subsequently followed by 'FOO unliked BAR'.

Does anybody know how I can locate the upvote by the current_user on a particular post and then destroy it?

Below is my controller code for like and unlike as it stands:

def like
  @ink.create_activity :like, owner: current_user
  @ink.upvote_by current_user

  redirect_to :back
end

def unlike
  @ink.downvote_by current_user

  redirect_to :back
end

Thanks

For anyone that stumbles across this in future, thanks to Element119 I was eventually able to pinpoint the current users likes on a particular post with a few variables and arel searches. I then destroyed the pinpointed likes.

def like
    @post.create_activity  :like, 
                           owner: current_user,
                           recipient: @post.user
    @post.upvote_by current_user

    redirect_to :back
  end



def unlike
    @post.downvote_by current_user

    @currentUserLikes = PublicActivity::Activity.where(trackable_id: @post.id, owner_id: current_user.id, key: "post.like")
    @currentUserLikes.destroy_all

    redirect_to :back
  end