Metallb L2Advertisement/IPAdressPools assignments behave strangely

I'm using metallb 0.13.4 L2, I have below IP advertisements and pools. Nginx grabs the right IP address and metallb speakers announce it properly. So IP addresses are correctly assigned.

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: external-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - external-pool
  nodeSelectors:
  - matchLabels:
      kubernetes.io/os: linux
      kubernetes.io/arch: amd64
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: internal-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - internal-pool
  nodeSelectors:
  - matchLabels:
      kubernetes.io/os: linux
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: external-pool
  namespace: metallb-system
spec:
  addresses:
    - x.x.x.204/32
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: internal-pool
  namespace: metallb-system
spec:
  addresses:
    - x.x.x.203/32

Nginx configs

....
  controller:
    annotations:
      metallb.universe.tf/address-pool: external-pool
....
---
....
  controller:
    annotations:
      metallb.universe.tf/address-pool: internal-pool
....

and from nginx controller events

Events:
  Type    Reason        Age                    From             Message
  ----    ------        ----                   ----             -------
  Normal  nodeAssigned  4m6s (x1173 over 19h)  metallb-speaker  announcing from node [redacted] with protocol "layer2"

See the (x1173 over 19h) so weird ? And when I look at the Ingresses their IP addresses change constantly but they are assigned to either internal or external nginx classes.

$ kl get ingressclass
NAME             CONTROLLER             PARAMETERS   AGE
nginx            k8s.io/ingress-nginx   <none>       5d6h
nginx-internal   k8s.io/ingress-nginx   <none>       5d6h

Although Ingress IPs constantly change between x.x.x.203 and x.x.x.204???, they always responds on the assigned IP address!!! This definitely looks very strange.

Note: I wasn't sure about the help in metallb project, that's why I'm creating the issue here.

The problem was the annotations on the controller, they should be at controller.service; Here is the working configuration;

controller:
  service:
    externalTrafficPolicy: Local
    type: LoadBalancer
    loadBalancerIP: x.x.x.203
    annotations:
      metallb.universe.tf/address-pool: "internal-pool"

Additionally, service must be type LoadBalancer and IP is specified.