AppEngine deploy cannot find Go packages

I have an AppEngine micro service setup in a monorepo, there is shared code between services so I've refactored to unify my go modules (they're really similar). The refactor works locally, builds and runs, and Goland compiles happily. My issue is that AppEngine deploy no longer works, receiving erorrs such as:

Error message: cmd/main.go:4:2: cannot find package "github.com/gin-gonic/gin" in any of:
        /usr/local/go/src/github.com/gin-gonic/gin (from $GOROOT)
        /layers/google.go.appengine_gopath/gopath/src/github.com/gin-gonic/gin (from $GOPATH)
cmd/main.go:5:2: cannot find package "mymodulename/customer/internal/mypkg" in any of:
        /usr/local/go/src/mymodulename/customer/internal/cauth (from $GOROOT)
        /layers/google.go.appengine_gopath/gopath/src/mymodulename/customer/internal/mypkg (from $GOPATH)

Original structure

    > svc1
      > cmd/main.go
      > internal
         >utils/shared.go
         >mypkg
      > go.mod
      > app.yaml
    > svc2
      > cmd/main.go
      > internal
         >utils/shared.go
         >mypkg
      > go.mod
      > app.yaml

After refactor

    > svc1
      > cmd/main.go
      > internal
         >mypkg
      > app.yaml
    > svc2
      > cmd/main.go
      > internal
         >mypkg
      > app.yaml
    > internal (common shared stuff)
      > utils/shared.go
    go.mod

Key points are utils/shared.go was moved to outside of each service directory and go.mod were unified.

What I'm unclear on is whether AppEngine builds the go binary on my local machine when i run glcoud app deploy or whether is bundles everything and runs it in cloud build.

1. How does AppEngine deploy work?
2. How do I get the AppEngine deploy to find my go.mod file?
3. How are the dependencies bundled? (if it's running on CloudBuild surely it doesnt have access to private repos)
   

The solution I've reached for anyone else who has the same issue. A few truths seem to exist, though the docs are somewhat ambiguous about it. The docs say:

Create your module's go.mod file in the same directory as your app.yaml file. App Engine searches the current directory, then successive parent directories until it finds a go.mod file.

But this does not seem to be true, infact, nothing seems to be copied at all above the app.yaml file.

So the solution requires:

1. Each microservice has a go.mod file of its own.
2. That go.mod file is in the same directory as the app.yaml
3. go mod edit is used to tell the Go compiler to look locally rather than try and fetch via the internet.
4. Vendoring is used to bundle all dependencies in the same directory as app.yaml so they're deployed to AppEngine.

A bit about local imports

Go appears to look for everything first in the dependency cache / paths, but then entirely on the internet. If I created my local package using go mod init shared, its module name is 'shared'. To tell Go that you want to import locally rather using the internet, invoke go mod edit -replace=shared=../../shared/, you should see your go.mod gets a line like replace shared => ../../shared. If you're using Goland and it still isn't compiling, try File>Invalidate Caches/Restart...

A bit about vendoring

go mod vendor in your go.mod folder will bundle all dependendies, including local ones so they can be deployed by AppEngine. This is also a good way to deal with private repos so you don't need to git Cloud Build access to your repo.