Incomplete Google Drive listings (API, desktop, drive.google.com activity) - can I force a refresh?

Minimal reproducible example

Using a browser, open a publicly shared google drive folder with files that are publicly accessible. This folder is added to the "Shared with me" section of Google Drive on drive.google.com.
Navigate to one of the folders with files if needed, and confirm that the files are listed.
a. Select the folder, check its activity list. It should not show activity for any of the files.
b. Make a files.list API call. It should not show any of the files.
'Open' one of the files (e.g. Preview it on drive.google.com or open it in a new tab.)
Repeat steps under 3, the file(s) 'opened' should now be listed in Activity/the API response.

I have been using the Google Drive API (v3) through the node.js API library to keep an eye on a "Shared With Me" folder (the folder is publicly shared), and alert of any new files. ( The files are publicly accessible; if you have the link to the file, you can access it, whether logged into Google services or not. )

However from time to time the resulting list (e.g. using files.list) is incomplete.

As an example, if the apparent 'folder structure' on drive.google.com is:

Shared Folder
  SubfolderA
    OldFile1.ext  
    OldFile2.ext
    NewFile1.ext  
    NewFile2.ext
  SunFolderB
    OldFile3.ext
    NewFile3.ext

Then the API might only return OldFile1.ext, OldFile2.ext, OldFile3.ext, and NewFile3.ext - but not NewFile1.ext nor NewFile2.ext.

Far be it for me to think my API use was without error, I double-checked other sources, and the files are also missing from:

The official API explorer's results
drive.google.com's Activity list
Google Drive Android APP's Activity list (including for the parent 'folder')
The (Windows) Desktop Google Drive virtual drive (streamed content)

This again despite being listed on drive.google.com when I open the parent 'folder'.

Stranger still, I can select a file and view its details, and all 4 of the above sources still insist the file doesn't exist. But if I access the actual content of the file, e.g. by opening a preview of the file in drive.google.com , lo and behold all 4 suddenly populate with that file, but only that file.
i.e. if I access the content of NewFile1.ext, that file pops up in the API, in the Activity List, and on the desktop, but NewFile2.ext is still nowhere to be found.

I would say what am I missing in my code, but as four official products seem to run into the same issue, I suppose my question is actually about what I am missing in terms of whether or not this is expected behavior, and how can I fix this without interacting with the google drive web interface.

Tried:
I tried the google drive API with files.list as is, as well as with a q = name contains 'NewFile' to see if that would help - but still only NewFile1.ext appeared (after I had 'opened' it in drive.google.com).

As mentioned, I also made sure to try the official API explorer, drive.google.com's activity list, the Android app's activity list for the parent 'folder' , nd the Google Drive for desktop app - none of which list (activity for) NewFile2.ext.

I tried increasing the pageSize and wound my way through every pageToken, and NewFile2.ext is simply nowhere to be found. Note that NewFile1.ext appeared on the first page of results with the original pageSize value of 10 after I 'opened' that file.

Expected:
For files.list to return every file, and not mysteriously decide to show some files only after 'opening' them in the Google Drive web interface.
( And relatedly, for the Google Drive for desktop to show every file, and for drive.google.com's Activity list to list every new file as an activity as well, without my first having to 'open' the file in the Google Drive web interface. )

tl;dr: Is there any way to force Google Drive products besides the drive.google.com web interface to realize new files exist?

Solution

Is there any way to force Google Drive products besides the drive.google.com web interface to realize new files exist?

Sharing the files to user accounts with Viewer access is one way.

From what I know, what you're getting is expected behavior. By default, files set to Anyone with the link that haven't been opened aren't fully accessible in the APIs, which is why nothing shows up in the Method: files.list of the user accounts.

I found Behaviour of "drive.file" scope on a shared folder on Google's Issue Tracker, which has a similar issue as yours. However, its current status is Won't fix (Obsolete), as this is intended.

Another one is Drive files.list returns inconsistent results when searching for files that were shared with an account via a group's member. Although this is currently Assigned, similar to the other ticket, it's likely the user hasn't opened the file shared to them, resulting in no results from the API. Feel free to +1 it if sharing with viewer permissions won't work for you or submit a feature request as this was set as a Bug.