I'm familiar with the Kubernetes quotas for CPU and memory usage. However, I'm envisaging a scenario in which certain containers are guaranteed to use a lot of network bandwidth, and I can't see any way of warning Kubernetes of this. (e.g. don't put two on the same machine, don't put too much else on this machine even if the other quotas are fine). How can I effect this sort of behaviour? For the purposes of the question I have full control of the cluster and am prepared to write code if necessary.
The only thing you can do is to apply Support traffic shaping using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. It can only be applied to your PODS.
Example:
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/ingress-bandwidth: 1M
kubernetes.io/egress-bandwidth: 1M
..
Also official k8s documentation gives link to bandwidth plugin. Try to apply in for your needs.
And read this GitHub issue.