If I may offer a suggestion: rather than playing around with the state
transitions, why not define some parameters setting the target goal for
page cache and buffer cache, and then take actions based on these goals?
For example, let's say you want 25% of memory in buffers and 40% in page
cache. At some point in time buffers are at 20% and page cache at 30%,
so you preferentially use the swap_out option to free memory.
If you had too much page cache and not enough buffers, you would look
for inode pages in shrink_mmap, but ignore buffer pages.
Sound reasonable? Shouldn't be too hard to implement ...
Regards,
Bill