I'm not very familiar with git-annex. I've just been reading the documentation trying to get my mind around it. The thought that has come to my mind, which I can't find certainty of is: Could I host a git repo on GitHub, and git push to it with git and git annex, but then set up git annex so that it pushes the binary files to like an S3 bucket, and all the source code files go to GitHub?
git-annex supports a variety of special remotes for storing archived content, including Amazon S3. It is possible to store git repository content (which includes symlinks to annexed content as well as any content stored directly by git alongside the annexed files, such as source code) whilst using special remotes (or other remotes where git-annex is available) to stored annexed content.