Is it possible to change system navigation to gesture or 3-button navigation programmatically?
This is not supported currently.
You can't change navigation in behalf of user. This is an important user preference and enabling apps to change the system navigation type would create confusion only. I don't think it will ever be supported.
Moreover there is no action here to even "guide" the user to the specific settings screen
On some devices this can be changed with adb commands (but there is no warranty those commands will work in future)
//Enable gesture navigation
adb shell cmd overlay enable com.android.internal.systemui.navbar.gestural
//Enable 2-button navigation
adb shell cmd overlay enable com.android.internal.systemui.navbar.twobutton
//Enable 3-button navigation
adb shell cmd overlay enable com.android.internal.systemui.navbar.threebutton