Using a SliverFillRemaining with a CustomScrollView and SliverList

I'm trying to create a scrolling list that can have a footer at the bottom of the scrolling list. If the list does not fill up all of the vertical screen space, the footer will need to shift down to the bottom of the page.

I tried implementing this with a SliverList and SliverFillRemaining in a CustomScrollView but the SliverFillRemaining shows some unexpected behavior I believe. It fills up more space than needed (see gif).

I used the following piece of code to create this list:

child: new CustomScrollView(
  slivers: <Widget>[
    new SliverList(
      delegate: new SliverChildBuilderDelegate(
        (BuildContext context, int index) {
          return new Card(
            child: new Container(
              padding: new EdgeInsets.all(20.0),
              child: new Center(child: new Text("Card $index")),
            ),
          );
        },
        childCount: 3,
      ),
    ),
    new SliverPadding(
      padding: new EdgeInsets.all(5.0),
    ),
    new SliverFillRemaining(
      child: new Container(
        color: Colors.red,
      ),
    )
  ],
)

SliverFillRemaining will automatically size itself to fill the space between the bottom of the last list item and the bottom of the viewport. See the performLayout method of SliverFillRemaining for the code:

https://github.com/flutter/flutter/blob/master/packages/flutter/lib/src/rendering/sliver_fill.dart#L118

I don't think you can use it to achieve the effect you're going for, though you might be able to create a subclass that would work.