My goal is to host web APIs in Azure using API Services using Azure Service Fabric. Azure Traffic Manager seemed to gear towards Infrastructure As A Service (IAAS) for its service such load balancing, fault tolerance etc. The Azure Service Fabric is geared towards Platform As A Service (PAAS) model with its own clustering. If my goal is to host web APIs (authored by leveraging Azure Service Fabric and without the need for Web Apps) in Azure, can I skip using Azure Traffic Manager because Service Fabric already provides the clustering? If not, then why should I use the Azure Traffic Manager? Am I missing something?
You are missing the fact that everything you have mentioned in there is reliant on a single region and that occasionally regions die.
If you have a Service Fabric located in West Europe for instance, and the West Europe region dies you have lost your entire solution.
If you have a Service Fabric in West Europe and North Europe, with data replicated, and a Traffic Manager profile spread across the two of them. In that instance when a region dies, you still have a working solution.
When building an available solution, you need to be able to point to any item within a solution and ask what happens if it fails. That should go from a VM to a whole region of datacenters.
Of course you need to weigh that against the cost of replicating your entire solution to another DC purely for reliance purposes. Region failures are very rare occurrences and you if you can handle some downtime you might be better simply ensuring your data is replicated to another region (by using GRS storage) and having a process to bring that data back online from another region.
But of course by doing that, you have made a decision about what you want to happen in the event of that failure. Which is the whole point.