I did :vert diffsplit file1.txt, then realized I didn't want to diff it but a different file. So I closed one window with :q and turned off the diff highlighting with :diffoff.
I then tried :vert diffsplit file2.txt, but now if I try to dp or do to put/get changes, I get the E101: More than two buffers in diff mode, don't know which one to use error.
How do I avoid this, or what am I doing wrong?
You should switch off diff mode for the unwanted buffer with :diffoff before quitting it with :q. Depending on your settings (e.g. :set hidden), the buffer will persist in the buffer list (:ls), and Vim will consider it when doing a diff.
A :bufdo diffoff should turn off diff for all existing buffers, though I usually just quit Vim and relaunch it with vimdiff ... should such a situation arise.