I would like ask you for help with git and how to split push to smaller one. I am not experienced in git so I need help.
My problem is that I can't send push bigger than 5GB to Azure DevOps it is reason why I need split push.
I have big push because I am trying reduce repository by removing binary files.
I follow these steps
git clone --bare --mirror
I am using git-filter-repo for reduce repository size.
after that
git reflog expire --expire=now --all
git repack -a -d --depth=250 --window=250 --max-pack-size=2g
Repository was reduced perfectly and packages are not bigger than 2GB. It does not help with push. because when I push my changes by
git push origin --force --mirror --prune
I get error about size.
I found solution with counting commits, but it does not work as I supposed. First it raised error that I am on mirror git. I do not know if I change something wrong way If I unset mirror flag from config. True is that founded code "worked", but it removed some branches. I need solution where I can push everything for all branches.
After my fail i found another code bellow, but I am afraid that it push only one branch (my be I am wrong). This code also need unset mirror flag.
# Split a repository into batches to avoid `pack exceeds maximum allowed size` on git push
REMOTE=origin
BRANCH=$(git rev-parse --abbrev-ref HEAD)
BATCH_SIZE=500
# check if the branch exists on the remote
if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then
# if so, only push the commits that are not on the remote already
range=$REMOTE/$BRANCH..HEAD
else
# else push all the commits
range=HEAD
fi
# count the number of commits to push
n=$(git log --first-parent --format=format:x $range | wc -l)
# push each batch
for i in $(seq $n -$BATCH_SIZE 1); do
# get the hash of the commit to push
h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1)
echo "Pushing $h..."
git push $REMOTE $h:refs/heads/$BRANCH
done
# push the final partial batch
git push $REMOTE HEAD:refs/heads/$BRANCH`
Please can you confirm me if this code does what I need. It means that pushes will be lower than 5GB and that push all my branches only reduced by filter-repo?
Thank you for your help
I used my split pushes two times in my script. First for backup repository to new one. Second for push changed commits from git filter-repo. I exctracted and rewrite my backup script and added here as solution.
[string] $RepositoryFolderPathForBareCloneBAK = "C:\Repositories\bare_clone"
[string] $RepositoryHttpsURL = "URLToRepository"
[string] $BackupRepositoryHttpsURL = "URLToBackupRepository"
[string] $remoteName = "origin"
[int] $maxPushSizeInMB = (5 * 1024) # 5GB - Azure DevOps limit for one Push
[int] $splitPushCommitsCount = 100
[boolean] $splitPush = $false
$ALocation = Get-Location
if ($null -ne (Test-Path $RepositoryFolderPathForBareCloneBAK)) {
write-host "Remove directory $RepositoryFolderPathForBareCloneBAK" -ForegroundColor Green
$null = Remove-Item $RepositoryFolderPathForBareCloneBAK -Recurse -Force -ErrorAction Stop | Out-Null
}
write-host "Clone bare repository from $RepositoryHttpsURL to $RepositoryFolderPathForBareCloneBAK" -ForegroundColor Green
git clone --bare --mirror $RepositoryHttpsURL $RepositoryFolderPathForBareCloneBAK
write-host "Backup repository" -ForegroundColor Magenta
write-host "Switch to folder $RepositoryFolderPathForBareCloneBAK" -ForegroundColor green
Set-Location $RepositoryFolderPathForBareCloneBAK -ErrorAction Stop
$doSplitPush = $splitPush #Split push if it is forced
if (!($doSplitPush)) {
# check if repository is lower than limit if not do split push
write-host "git count-objects -vH" -ForegroundColor Green
$gitCountResult = (git count-objects -vH)
$splitRegEx = "^(?<Name>[^\s-]+(-[^\s-]+)*):\s*(?<Value>\d+\.?\d*)\s*(?<Measure>\S+)?$"
$repositorySize = 0
$gitCountResult | ForEach-Object {
write-host $_ -ForegroundColor Green
if ($_ -match $splitRegEx) {
if ($matches['Name'] -eq 'size-pack') {
if ("$($matches['Measure'])" -ne '') {
if (($matches['Measure']) -eq 'bytes') {
$repositorySize = ([math]::Round((([float]($matches['Value'])) / 1024 / 1024)))
}
elseif (($matches['Measure']) -eq 'KiB') {
$repositorySize = ([math]::Round(([float]($matches['Value']) / 1024)))
}
elseif (($matches['Measure']) -eq 'MiB') {
$repositorySize = ([float]($matches['Value']))
}
elseif (($matches['Measure']) -eq 'GiB') {
$repositorySize = (([float]($matches['Value'])) * 1024)
}
else {
$repositorySize = ([float]($matches['Value']))
}
}
}
}
}
$doSplitPush = ($repositorySize -ge $maxPushSizeInMB)
}
if (!($doSplitPush)) {
write-host "Push all to remote repository" -ForegroundColor Magenta
write-host "git push --mirror $BackupRepositoryHttpsURL" -ForegroundColor Green
git push --mirror $BackupRepositoryHttpsURL
}
else {
write-host "Splited git pushes to remote repository. Push per $($splitPushCommitsCount) commits" -ForegroundColor Magenta
if ((git config remote.origin.mirror) -eq "true") {
write-host "git config --unset remote.origin.mirror" -ForegroundColor Green
git config --unset remote.origin.mirror
}
# Split a repository into batches to avoid `pack exceeds maximum allowed size` on git push
$REMOTE = $remoteName
$NewREMOTE = "new_$($remoteName)"
$BATCH_SIZE = $splitPushCommitsCount
write-host "Get list of remotes" -ForegroundColor green
$Remotes = git remote -v
if ($Remotes) {
$RemoteExtists = ($null -ne ($Remotes | Where-Object { $_.IndexOf($BackupRepositoryHttpsURL) -ne -1 } ))
if (!$RemoteExtists) {
if ($null -ne ($Remotes | Where-Object { $_.IndexOf($NewREMOTE) -ne -1 } )) {
write-host "Set existing remote $($NewREMOTE) for $($BackupRepositoryHttpsURL)" -ForegroundColor green
git remote set-url "$NewREMOTE" "$BackupRepositoryHttpsURL" | Out-Null
}
else {
write-host "Add remote $NewREMOTE for $BackupRepositoryHttpsURL" -ForegroundColor green
git remote add "$NewREMOTE" "$BackupRepositoryHttpsURL" | Out-Null
}
}
}
git for-each-ref --format="%(refname)" --sort='authordate' | ForEach-Object {
if ($_.IndexOf("refs/heads/") -ne -1) {
$BRANCH = $_.Replace("refs/heads/", '')
write-host "Pushing branch $($BRANCH)" -ForegroundColor yellow
write-host "Switch $BRANCH" -ForegroundColor green
write-host "git symbolic-ref HEAD $($_)" -ForegroundColor green
git symbolic-ref HEAD $_ | Out-Null
# check if the branch exists on the remote
if ($null -ne (git show-ref --quiet --verify "refs/remotes/$($NewREMOTE)/$($BRANCH)")) {
# check if the branch exists on the remote
$range = "$($NewREMOTE)/$($BRANCH)..HEAD"
}
else {
# else push all the commits
$range = "HEAD"
}
write-host "Range $($range)" -ForegroundColor green
# count the number of commits to push
$n = (git log --first-parent --format="format:x $range").Length
write-Host "Count of commits to push $($n)" -ForegroundColor green
if ($n -gt 0) {
$loopCount = [int] [Math]::Truncate($n / $BATCH_SIZE)
# push each batch
for ($i = 1; $i -le $loopCount; $i++) {
$h = git log --first-parent --reverse --format=format:%H --skip ($n - ($i * $BATCH_SIZE)) -n1
write-Host "Skip commits $(($n - ($i * $BATCH_SIZE))) to $($h)" -ForegroundColor green
Write-Host "Pushing $($NewREMOTE) --force $($h):refs/heads/$($BRANCH)"
git push "$($NewREMOTE)" --force "$($h):refs/heads/$($BRANCH)"
}
# push last batch
Write-Host "Pushing $($NewREMOTE)" --force "HEAD:refs/heads/$($BRANCH)"
git push "$($NewREMOTE)" --force "HEAD:refs/heads/$($BRANCH)"
}
else {
write-host "Skiped push no commits" -ForegroundColor Yellow
}
}
}
write-host "git push $NewREMOTE --force 'refs/tags/*'" -ForegroundColor Green
git push $NewREMOTE --force 'refs/tags/*'
write-host "git push $NewREMOTE --force 'refs/replace/*'" -ForegroundColor Green
git push $NewREMOTE --force 'refs/replace/*'
}
Set-Location $ALocation
if (Test-Path -Path $RepositoryFolderPathForBareCloneBAK) {
write-host "Remove directory $RepositoryFolderPathForBareCloneBAK" -ForegroundColor Green
$null = Remove-Item $RepositoryFolderPathForBareCloneBAK -Recurse -Force -ErrorAction Stop | Out-Null
}