KRaft design architecture - Separation of Controllers and Brokers

We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )

Since we are dealing with Kafka production cluster , we want to understand what is the best Practice and the right approach to install the machines and configuration

The question about the cluster that we are preparing is about the controllers

Do we need to set and install the controllers on dedicated machines and not on Kafka machines itself?

for example I gives here PLAN1 and PLAN2

PLAN1

We can install 3 controllers on the first 3 Kafka machines ( means both PID's are running on Kafka machine as broker pid and controller pid [ pid - Linux process id ]

kafka1-3 will include pid for broker and pid for controller

kafka4+ will include only pid for broker

PLAN2

We can install 3 controllers on different dedicated machines

35 physical machines for Kafka Kraft service

3 dedicated VM machines for Kafka controllers ( VM machines since from our understanding controllers are not consume so much resources )

The second question , is about how many controllers services we need?

Is it always 3 controllers for 3 Kafka machines or for 15 Kafka machines or for 35 Kafka machines ?

Think about it like this - Raft is a leader election protocol. If you lost 1/3 of the nodes; you're left with 2 equal "electors" who may not agree on a decision.

So, having more controllers is better for quorum, but you generally don't need more than 5 or 7 (odd numbers recommended). This didn't really change from Zookeeper or Kraft...

Two "pid" on any node isn't what you want. process.roles accepts both roles of broker and controller, and would run as a single Java process. You can also separate them, and ultimately is determined by if you have the hardware available and how many nodes you're willing to lose