Checking the latest image used in the metrics-server Github repo, the tag used is v0.5.0, for arm64 I would usually add arm64 to the image name and pull it.
But the image doesn't exist and doing an inspect to the base image shows that its arch is amd64.
In google's registry the latest image is v0.3.6. So I'm not sure if support for arm64 has continued or staled.
There's no need to append arm64 starting v0.3.7, the image support multiple architectures. See official FAQ here with complete image url.