How to prevent empty bars from taking up width in Chart.js bar chart

I'm building a bar chart in Chart.js. I have multiple data series, but often, not all of them will have a value. When this happens, even though the bars aren't there, Chart.js seems to want to allocate space for them:

Notice how the green and blue bars above are always left aligned relative to the labels, and how the pink and yellow are always on the right. How do I make the bars appear centered relative to their labels?

@Drarig29's answer is correct -- null values plus skipNull removes the extra bars. However, if you're using barPercentage, your bars will fluctuate in width (varies from being a 1/5th of the width to 100% of the width, in my case, depending on how many bars there are), so you must set a maxBarThickness on the dataset to prevent them from growing.