Using a shadow together with backdrop-filter: blur

I've been experimenting with backdrop-filter lately, usually using it for blurring whatever was behind an element (which is dynamic, so I can't just use things like this). However, I also needed to apply a shadow to said element, and so I simply added box-shadow: /* something, not inset */.

Unfortunately, as a result, the blur effect was extended to all of the area covered by the shadow (which seems logical, as it's called backdrop filter). You can see a demo of it below (note that you will need a browser that supports backdrop-filter, in case that wasn't already obvious).

#background {
  position: absolute;
  
  width: 600px;
  height: 300px;
  z-index: -1;
  
  background-image: url('https://picsum.photos/600/300/');
  background-repeat: no-repeat;
}

#blurryandshadowy {
  display: inline-block;

  margin: 50px;
  padding: 50px;
  
  background: rgba(255, 255, 255, 0.25);
  backdrop-filter: blur(15px);
  box-shadow: 0 12px 24px rgba(0, 0, 0, 0.25);
}

<div id="background"></div>
<div id="blurryandshadowy">
  . . .
  <br>. . .
</div>

Is there a way to apply a shadow that doesn't conflict (is conflict even a verb?) with backdrop filters? If necessary, I could also use JavaScript (which is why it's within this post's tags) — Of course, a CSS-only answer would be better appreciated.

EDIT: works correctly in 2020 🙃

Solution

You can probably have two different layers using pseudo element. One layer for the filter and the other for the shadow:

#background {
  position: absolute;
  width: 600px;
  height: 300px;
  z-index: -1;
  background-image: url('https://lipsum.app/600x300/');
  background-repeat: no-repeat;
}

#blurryandshadowy {
  display: inline-block;
  margin: 50px;
  padding: 50px;
  position: relative;
  z-index: 0;
}

#blurryandshadowy:before {
  content: "";
  position: absolute;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
  background: rgba(255, 255, 255, 0.25);
  backdrop-filter: blur(20px);
  -webkit-backdrop-filter: blur(20px);
  z-index: 1;
}

#blurryandshadowy:after {
  content: "";
  position: absolute;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
  box-shadow: 0 12px 24px rgba(0, 0, 0, 0.25);
}

<div id="background"></div>
<div id="blurryandshadowy">
  . . .
  <br>. . .
</div>