SwipeRefreshLayout blocking horizontally scrolled RecyclerView

My setup is simple enough:

<android.support.v4.widget.SwipeRefreshLayout
     android:id="@+id/swiperefresh"
     android:layout_width="match_parent"
     android:layout_height="match_parent" >

     <android.support.v7.widget.RecyclerView
         android:id="@+id/recyclerView"
         android:layout_width="match_parent"
         android:layout_height="220dp"/>

</android.support.v4.widget.SwipeRefreshLayout>

The content of onCreate():

layoutManager = new LinearLayoutManager( this );
layoutManager.setOrientation( LinearLayoutManager.HORIZONTAL );
topTopicRecyclerView.setLayoutManager( layoutManager );

Now, when I swipe the recyclerView left or right and the swipe angle is not perfectly horizontal, the SwipeRefreshLayout jumps in and takes over the scrolling control. That leads to annoying visual "hiccups" inside the recyclerView.

If the SwipeRefreshLayout is disabled, all is fine.

So, how can I deactivate the SwipeRefreshLayout's scrolling control over the RecyclerView's area?

As per this discussion about SRL and HorizontalScrollView, I created the counterpart for the SwipeRefreshLayout:

public class OnlyVerticalSwipeRefreshLayout extends SwipeRefreshLayout {

  private int touchSlop;
  private float prevX;
  private boolean declined;

  public OnlyVerticalSwipeRefreshLayout( Context context, AttributeSet attrs ) {
    super( context, attrs );
    touchSlop = ViewConfiguration.get( context ).getScaledTouchSlop();
  }

  @Override
  public boolean onInterceptTouchEvent( MotionEvent event ) {
    switch( event.getAction() ){
      case MotionEvent.ACTION_DOWN:
        prevX = MotionEvent.obtain( event ).getX();
        declined = false; // New action
        break;

      case MotionEvent.ACTION_MOVE:
        final float eventX = event.getX();
        float xDiff = Math.abs( eventX - prevX );
        if( declined || xDiff > touchSlop ){
          declined = true; // Memorize
          return false;
        }
        break;
    }
    return super.onInterceptTouchEvent( event );
  }
}

and usage in XML:

<com.commons.android.OnlyVerticalSwipeRefreshLayout
     android:id="@+id/swiperefresh"
     android:layout_width="match_parent"
     android:layout_height="match_parent" >

   <tags/>

</com.commons.android.OnlyVerticalSwipeRefreshLayout>