Make anchor link go some pixels above where it's linked to

I'm not sure the best way to ask/search for this question:

When you click on an anchor link, it brings you to that section of the page with the linked-to area now at the VERY TOP of the page. I would like the anchor link to send me to that part of the page, but I would like some space at the top. As in, I don't want it to send me to the linked-to part with it at the VERY TOP, I would like 100 or so pixels of space there.

Does this make sense? Is this possible?

Edited to show code - it's just an anchor tag:

<a href="#anchor">Click me!</a>

<p id="anchor">I should be 100px below where I currently am!</p>

UPDATE: No more upvotes, please. Even though this is the accepted answer, the better and more modern answer is this one from user127091. My answer here has become quite outdated (unless you really want to use javascript).

window.addEventListener("hashchange", function () {
    window.scrollTo(window.scrollX, window.scrollY - 100);
});

This will allow the browser to do the work of jumping to the anchor for us and then we will use that position to offset from.

EDIT 1:

As was pointed out by @erb, this only works if you are on the page while the hash is changed. Entering the page with a #something already in the URL does not work with the above code. Here is another version to handle that:

// The function actually applying the offset
function offsetAnchor() {
    if(location.hash.length !== 0) {
        window.scrollTo(window.scrollX, window.scrollY - 100);
    }
}

// This will capture hash changes while on the page
window.addEventListener("hashchange", offsetAnchor);

// This is here so that when you enter the page with a hash,
// it can provide the offset in that case too. Having a timeout
// seems necessary to allow the browser to jump to the anchor first.
window.setTimeout(offsetAnchor, 1); // The delay of 1 is arbitrary and may not always work right (although it did in my testing).

NOTE: To use jQuery, you could just replace window.addEventListener with $(window).on in the examples. Thanks @Neon.

EDIT 2:

As pointed out by a few, the above will fail if you click on the same anchor link two or more times in a row because there is no hashchange event to force the offset.

This solution is very slightly modified version of the suggestion from @Mave and uses jQuery selectors for simplicity

// The function actually applying the offset
function offsetAnchor() {
  if (location.hash.length !== 0) {
    window.scrollTo(window.scrollX, window.scrollY - 100);
  }
}

// Captures click events of all <a> elements with href starting with #
$(document).on('click', 'a[href^="#"]', function(event) {
  // Click events are captured before hashchanges. Timeout
  // causes offsetAnchor to be called after the page jump.
  window.setTimeout(function() {
    offsetAnchor();
  }, 0);
});

// Set the offset when entering page with hash present in the url
window.setTimeout(offsetAnchor, 0);

JSFiddle for this example is here