i wanted to setup my local mainline branch to track origin/mainline but used the command
git branch --set-upstream origin/mainline mainline by mistake,
with the outcome Branch origin/mainline set up to track local branch mainline.
How can I fix this so local mainline tracks remote mainline?
You have the wrong arguments order, what you want is git branch --set-upstream mainline origin/mainline.
see:
-t, --track
When creating a new branch, set up configuration to mark the start-point branch as "upstream" from the new branch. This configuration will tell git to show the relationship between the two branches in git status and git branch -v. Furthermore, it directs git pull without arguments to pull from the upstream when the new branch is checked out.
This behavior is the default when the start point is a remote-tracking branch. Set the branch.autosetupmerge configuration variable to false if you want git checkout and git branch to always behave as if --no-track were given. Set it to always if you want this behavior when the start-point is either a local or remote-tracking branch.
--set-upstream
If specified branch does not exist yet or if --force has been given, acts exactly like --track. Otherwise sets up configuration like --track would when creating the branch, except that where branch points to is not changed.
So what your had did is creating a local branch named origin/mainline which tracks mainline branch.