Why do I need to use Build Agents in a CICD pipeline?

This seems to be such a basic questions that I've failed to find a good answer anywhere.
Why can't I use my CICD-system to do all the tasks needed by itself?
Do I really have to set up build agents?
What would the main common disadvantages/risks be of not using build agents?

In this case, my question concerns MS Azure DevOps Services/Server.

What I can tell, build agents are needed for at least the following reasons:

• Make sure the system used for builds only have known components installed to avoid the "but it works on my computer"-syndrome ;)
• Off load the system managing the pipeline, a build can be very CPU, network, disk and database intensive
• Provide a security boundary between projects with regards to what a build server can access, execute and configure to keep secret and/or sensitive projects secret
• And, of course. Some settings in the build environment is project specific and would disrupt builds of other projects if only using one setup.