How do I prevent onTapDown from being triggered on a parent widgets GestureDetector?

I have a Stack in which several widget can be dragged around. In addition, the container that the Stack is in has a GestureDetector to trigger on onTapDown and onTapUp. I want those onTap events only to be triggered when the user taps outside of the widget in the Stack. I've tried the following code:

class Gestures extends StatefulWidget {
  @override
  State<StatefulWidget> createState() => _GesturesState();
}

class _GesturesState extends State<Gestures> {
  Color background;
  Offset pos;

  @override
  void initState() {
    super.initState();
    pos = Offset(10.0, 10.0);
  }

  @override
  Widget build(BuildContext context) => GestureDetector(
    onTapDown: (_) => setState(() => background = Colors.green),
    onTapUp: (_) => setState(() => background = Colors.grey),
    onTapCancel: () => setState(() => background = Colors.grey),
        child: Container(
          color: background,
          child: Stack(
            children: <Widget>[
              Positioned(
                top: pos.dy,
                left: pos.dx,
                child: GestureDetector(
                  behavior: HitTestBehavior.opaque,
                  onPanUpdate: _onPanUpdate,
//                  onTapDown: (_) {},  Doesn't affect the problem
                  child: Container(
                    width: 30.0,
                    height: 30.0,
                    color: Colors.red,
                  ),
                ),
              )
            ],
          ),
        ),
      );

  void _onPanUpdate(DragUpdateDetails details) {
    RenderBox renderBox = context.findRenderObject();
    setState(() {
      pos = renderBox.globalToLocal(details.globalPosition);
    });
  }
}

However, when starting to drag the widget, the onTap of the outermost container is triggered as well, making the background momentarily go green in this case. Settings HitTestBehavior.opaque doesn't seem to work like I'd expect. Neither does adding a handler for onTapDown to the widget in the Stack.

So, how do I prevent onTapDown from being triggered on the outermost GestureDetector when the user interacts with the widget inside of the Stack?

Update:

An even simpler example of the problem I'm encountering:

GestureDetector(
  onTapDown: (_) {
    print("Green");
  },
  child: Container(
    color: Colors.green,
    width: 300.0,
    height: 300.0,
    child: Center(
      child: GestureDetector(
        behavior: HitTestBehavior.opaque,
        onTapDown: (_) {
          print("Red");
        },
        child: Container(
          color: Colors.red,
          width: 50.0,
          height: 50.0,
        ),
      ),
    ),
  ),
);

When I tap and hold the red container, both "Red" and "Green" are printed even though the inner GestureDetector has HitTestBehavior.opaque.

I found that I could get the behavior that I wanted (making a widget appear transparent to it parent, while still responding to pointer events) by creating a render object with hit test behavior like this:

  @override
  bool hitTest(BoxHitTestResult result, {@required Offset position}) {
    // forward hits to our child:
    final hit = super.hitTest(result, position: position);
    // but report to our parent that we are not hit when `transparent` is true:
    return false;
  }

I've published a package with a widget having this behavior here: https://pub.dev/packages/transparent_pointer.