I think the best way to delete a repository is to first delete the ".git" hidden folder then, if needed remove other files and direcotories. But, why removing all in once take so long ?
Basically, git stores all revisions of your code, that means, each time you do a commit git copy the whole project somewhere under .git/.
Knowing that, you will easily understand how it could become long to remove this folder.
Here is a pretty good SO post about that: How does git store files?