This article might help you to think about how you manage interactions between virtual hands and object. Rather than preventing penetration another method is to look at how you can use the penetration to show the user what is happening and increase the sense of presence.