Wrong back button behavior when using setOnItemSelectedListener when using it on BottomNavigationView android

By default in Android, when using the BottomNavigationView, navigation by pressing on item looks like this:

A -> B -> C

Back button: C -> A

But when using setOnItemSelectedListener, the navigation breaks and when you click on the back button it looks completely different:

A -> B -> C

Back button: C -> B -> A

How can I fix this and make it so that when the back button is clicked, the navigation always leads to the first (startDestination) item?

My code:

bottomNavigationView.setOnItemSelectedListener {
    when(it.itemId) {
        R.id.homeFragment -> { navController.navigate(R.id.homeFragment) }
        R.id.favoriteFragment -> { navController.navigate(R.id.favoriteFragment) }
        R.id.profileFragment -> { navController.navigate(R.id.profileFragment) }
    }
    true
}

menu.setOnNavigationItemSelectedListener {
        when(it.itemId){
            R.id.homeFragment -> {
                val graph = navController.graph
                graph.startDestination = R.id.homeFragment
                navController.graph = graph
            }
    }
}

But that's the best menu.setupWithNavController(navController)