I understand that the working set is cached every time a query is run in the mongoDB.
What happens if one working set exceeds the caching memory when I know that the data on the previous page is removed and cached?
Ex) cacheSizeGB: 0.5g, total document size about query: 1g
And is it helpful to reduce the size of the document being cached by using the project command?
The cache is managed by WiredTiger, the mongod will request documents it needs, and if WT doesn't find it in the cache, it will read it in from disk. If this makes the cache exceed the threshold (default 80% of maximum size), the background eviction workers will start removing the least recently used items from the cache.
The result set is constructed/sorted in heap memory, not cache. If your result set is too large, mongod may be able to use disk for temporary space while sorting.
Large results may also cause the operating system to use swap, and in extreme cases it might run out of memory and get killed by the OOM killer.