After reading google recommendation on using the Vary header on mobile sites that deliver different html/content for the same url, I was trying to figure out what happens if you have a new mobile site for example and it only covers some parts of your site, leaving some pages delivering the same html/content for desktop as well as mobile devices. Do I need to still add the Vary header for all pages, regardless whether the html/content do actually vary by user agent or should I add the header only to those urls that do deliver different html/content for desktop and for mobile devices. Will google penalise the site if the vary header is found on a certain page that doesn't "vary by user agent" (while other pages of the same site do)?
You're pretty unlikely to suffer any kind of penalty for this. I can't imagine that they're going to hammer a site that offers a UA header when the content doesn't vary on a few pages; you're not trying to actively deceive anyone.
The only time where this might be an issue is if your user-agent shows significantly different copy on page to a search engine user-agent than it did to a browser or mobile user agent and it deemed the Vary header was being abused to do so. That might cause a penalty.
Essentially, work in good faith and never "cloak" any content (I know that's not what you're intending, but just make sure that the search engine spiders aren't being deceived) and you will be fine.