I was able to restrict users from editing pipeline but now I would also like to remove possibility to change branch when queuing a build. So that they can for example build only the master branch and no other.
I don't think it's possible, as a workaround you can add a small PowerShell script that checks the branch and if it's not the master fail the build:
if ("$(Build.SourceBranch)" -ne "refs/heads/master")
{
Write-Error "The branch is not the master!"
}