Using cURL_multi is great for simultaneous connections, but what can limit the number of connections? I ask because when doing over 500 some of them seem to randomly fail to connect. I am doing HEAD requests with a timeout of 5, and increasing the timeout to 10 didn't seem to help. Any ideas?
500 concurrent connections? That's a lot to ask for, the server is probably refusing to serve you.
Try setting a more reasonable limit, between 4-12 concurrent connections.