SliverHeader behaviour management

My flutter app has a NestedScrollView with a headerSliverBuilder.

My SliverAppBar inside this builder is like this:

SliverAppBar(
    elevation: 0,
    snap: true,
    pinned: false,
    floating: true,
    forceElevated: false,
    primary: false,
    automaticallyImplyLeading: false,
    backgroundColor: Colors.white,
    expandedHeight: 65.0 + 75,
    flexibleSpace: Container(
      child: Column(children: <Widget>[
        MyTabBar(true, 65, () {}, () {}),
        Container(height: 75, color: Colors.orange)
      ]),
      decoration: BoxDecoration(boxShadow: [
        BoxShadow(
          color: Colors.black26,
          blurRadius: 4.0,
          spreadRadius: .0,
        ),
      ], color: Colors.white),
    )
)

Now, when scrolling the behaviour is kind of weird. I'd like it to scroll away the whole SliverAppBar including everything in flexibleSpace. But this is what happens:

It scrolls underneath the Orange part of my header and then scrolls away the tab-part above. And the moment the tab-part disappears, the orange part vanishes too. Huhh..

And the same when I get the bar back from scrolling down. Then the orange part just appears on the spot and the top part slides in. I'd like to have it all sliding in.

The whole behavior changed for me since the last flutter update.

Does anyone know what I need to do different to achieve my expected behavior?

Here is my full code of this simple example: https://github.com/nietsmmar/SliverHeaderTest/tree/master/flutter_app

How about using AnimatedContainer inside a Stack widget?

Something like this...

import 'package:flutter/material.dart';
import 'package:vector_math/vector_math_64.dart' hide Colors;

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final ScrollController scrollController = ScrollController();

    return MaterialApp(
      theme: ThemeData(
        primaryColor: Colors.blue,
      ),
      home: Scaffold(
        body: Stack(
          children: <Widget>[
            SafeArea(
              child: ListView(
                controller: scrollController,
                padding: const EdgeInsets.only(top: 100),
                children: List.generate(
                  100,
                  (index) => ListTile(
                    title: Text('This is item #$index'),
                  ),
                ),
              ),
            ),
            AnimatedAppBar(
              scrollController: scrollController,
              child: PreferredSize(
                preferredSize: Size.fromHeight(100),
                child: const Center(child: Text('APP BAR')),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

class AnimatedAppBar extends StatefulWidget {
  AnimatedAppBar({
    required this.scrollController,
    this.child,
    this.toolbarHeight,
  });

  final ScrollController scrollController;
  final PreferredSize? child;
  final double? toolbarHeight;

  @override
  _AnimatedAppBarState createState() => _AnimatedAppBarState();
}

class _AnimatedAppBarState extends State<AnimatedAppBar> {
  late double _totalHeight;
  late double _lastPosition;
  late bool _isShown;

  @override
  void initState() {
    super.initState();

    _totalHeight = widget.toolbarHeight ??
        kToolbarHeight + (widget.child?.preferredSize.height ?? 0.0);
    _lastPosition = widget.scrollController.offset;
    _isShown = true;

    widget.scrollController.addListener(_scrollListener);
  }

  void _scrollListener() {
    if (_isShown && _lastPosition < widget.scrollController.offset) {
      setState(() => _isShown = false);
    } else if (!_isShown && _lastPosition > widget.scrollController.offset) {
      setState(() => _isShown = true);
    }
    _lastPosition = widget.scrollController.offset;
  }

  @override
  Widget build(BuildContext context) {
    return AnimatedContainer(
      width: double.infinity,
      height: _totalHeight,
      padding: EdgeInsets.only(top: kToolbarHeight),
      duration: const Duration(milliseconds: 250),
      curve: Curves.easeIn,
      transform: Matrix4.translation(
        Vector3(0, _isShown ? 0 : -_totalHeight, 0),
      ),
      color: Theme.of(context).primaryColor,
      child: widget.child,
    );
  }

  @override
  void dispose() {
    widget.scrollController.removeListener(_scrollListener);
    super.dispose();
  }
}