Back button closes app instead of going to previous fragment android navigation component

While creating a very simple sample app, I couldn't wrap my head around why my app is closing when I press the hardware back button on my emulator.

I have 1 mainActivity and 2 fragments.

When I am on the NavigationFragment and press back, the app closes instead of going back to IntermediateFragment.

MainActivity:

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        toolbar.setTitle(R.string.app_name)
    }
}

activity_main.xml:

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
xmlns:app="http://schemas.android.com/apk/res-auto"
tools:context="com.exampleapplication.MainActivity">

<androidx.appcompat.widget.Toolbar
    android:id="@+id/toolbar"
    android:layout_width="match_parent"
    android:layout_height="?attr/actionBarSize"
    app:layout_constraintTop_toTopOf="parent"
    />

<fragment
    android:id="@+id/my_nav_host_fragment"
    android:name="androidx.navigation.fragment.NavHostFragment"
    app:navGraph="@navigation/main_nav"
    android:layout_width="match_parent"
    android:layout_height="0dp"
    app:layout_constraintTop_toBottomOf="@+id/toolbar"
    app:layout_constraintBottom_toBottomOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>

navigation_graph:

<?xml version="1.0" encoding="utf-8"?>
<navigation xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:id="@+id/main_nav"
    app:startDestination="@+id/intermediateFragment">

    <fragment
        android:id="@+id/intermediateFragment"
        android:name="com.exampleapplication.IntermediateFragment">
        <action
            android:id="@+id/action_intermediate_to_navigation"
            app:destination="@+id/navigationFragment"
            />
    </fragment>

    <fragment
        android:id="@+id/navigationFragment"
        android:name="com.exampleapplication.NavigationFragment"
        />
</navigation>

IntermediateFragment:

class IntermediateFragment : Fragment() {
    override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
        return inflater.inflate(R.layout.fragment_intermediate, container, false)
    }

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        btn_next_fragment.setOnClickListener {
            findNavController().navigate(R.id.action_intermediate_to_navigation)
        }
    }
}

NavigationFragment:

class NavigationFragment : Fragment() {

    override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
        return inflater.inflate(R.layout.fragment_navigation, container, false)
    }

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        btn_first_library.setOnClickListener {
            findNavController().setGraph(R.navigation.first_library_nav)
        }

        btn_download_pdf.setOnClickListener {
            findNavController().setGraph(R.navigation.download_pdf_nav)
        }
    }
}

Any ideas?

You're missing one line on your <fragment>:

app:defaultNavHost="true"

As per the Navigation Getting Started guide:

The app:defaultNavHost="true" attribute ensures that your NavHostFragment intercepts the system Back button.

Since you don't set that, Navigtion does not intercept the back button and hence, you only get the default activity behavior (which is closing your activity).