If I add a lot of comments to my site, will it increase the load time for a user that doesn't have a fast internet connection?
Comments do not have a negative impact on the loading time (the browser ignore it), only on the downloading time but is not significant.
If you have problems with load times the are other factor you can improve:
A good tool for analyze and optimization web pages is google pagespeed