Google has introduced some changes recently related to storage APIs in API 29 like scoped storage and we opted out by adding 'requestLegacyExternalStorage=true' in Manifest. But now when I targetSdkVersion 30, this no longer seems to work. Some of the files in the download directories were not listing (File.listFiles) after this change.
But now when I targetSdkVersion 30, this no longer seems to work
That is correct. Android 11 (API 30+) requestLegacyExternalStorage=true does nothing and you can no longer "opt-out". It was available in Android 10 to give developers a transition/grace period to be able to migrate to the scoped storage model.
Option 1: Migrate data in your app whilst still targeting API 29, then once you're migrated data is compatible with scoped storage you should be able to release an update targetting API 30 - https://developer.android.com/training/data-storage/use-cases
This can come with its own problems if users skip this version and updates directly from a previous version to the latest and you're stuck with un-migrated data you can't access.
Option 2: It seems that Google sees this obvious caveat and has included a preserveLegacyExternalStorage=true option when targetting API 30 to allow you to migrate data. https://developer.android.com/reference/android/R.attr#preserveLegacyExternalStorage
Going forward you can reference this table for deciding what storage "framework" to use based on the use-case: https://developer.android.com/training/data-storage
There is a potential that some apps simply won't be able to successfully migrate, based on how they interacted with the File API as Google's solution will not encompass all current use-cases and there might not be a migration path.
For instance, I released an app a couple of years ago that allowed users to update album artwork using MediaStore and ContentResolver to update the data for the album artwork image - this was stored in shared storage. Having looked at the Android 10+ AOSP MediaProvider source code it seems that apps that used to use MediaStore to update album artwork to point to a data file no longer works, simply because the MediaProvider internally creates its own artwork in a hidden .thumbnails folder looking directly at the mp3's and using a MediaExtractor, and never references the ContentValues that were inserted to reference the artwork. So even though you can update the artwork yourself, query the MediaStore for it and see it, other apps have to use ContentResolver#loadThumbnail in API 29+ that does not reference your updated values and either creates an artwork lazily, or picks the already generated file in the .thumbnails folder. Obviously, none of this is documented, and I got a massive backlash to my app with negative reviews, yet these changes were breaking changes and completely out of my control and took me looking through AOSP source code to find that Android had fundamentally changed behaviour.
(This wasn't a rant, but an example of how these changes offered no migration path because of fundamental undocumented behaviour to AOSP).