Flutter slider should scroll only when dragged with thumb bar

In my custom widgets, there are multiple components along with the scrolling of page. Whenever I accidentally touch nearby to a slider by, it is recognizing touch & changing the position to respective position. I need custom behavior so that Slider changes only when it is dragged with a thumb bar.

I tried by adding a gesture recognizer on top of the slider but gives the flicker effect of changing & resetting value. Is it possible so that Slider shouldn't react on touch other than drag?

 GestureDetector(
        onHorizontalDragUpdate: (updateDetails) {
          print("On onHorizontalDragUpdate called");
          setState(() {
            _diffLevel = updateDetails.globalPosition.distance;
          });
        },
        onTap: (() {
          setState(() {
            _diffLevel = _diffLevel;
          });
          print("On Tap called");
        }),
        child: Slider.adaptive(
              value: _diffLevel.toDouble(),
              activeColor: MyColors.appMediumBlueColor(),
              //inactiveColor: Colors.purple,
              onChanged: (double newValue) {
                setState(() {
                  _diffLevel = newValue;
                });
              },
              min: widget.minLevel,
              max: widget.maxLevel,
              divisions: widget.divisions,
              label: _diffLevel.toString(),
            ),
      )

Expected Result :

Actual result:

If I understand correctly, you are looking for a slider implementation that allows only the drag without the jumping. That means, the slider should move only to nearby values and shouldn't jump to values that are further away.

This can easily be handled in the onchanged. From your code, you are saving the slider value in a global variable _diffLevel. Let's say, your each division is 50.

So, your onChanged could be this:

onChanged: (double newValue) {
                  setState(() {
                    double diff = (newValue - _diffLevel).abs();
                    if(diff<=50) {
                      _diffLevel = newValue;
                    }
                  });
                },

This constraints the slider to change its value only when the difference between its current value and new value is within the range of 50 ie. in range of 50 around the current value. So when you click further way, it wont change. It will only change when you slide it to the next division or the previous division.