As in text books OS will move dirty pages to swap space based on LRU algorithm and make space for new process in RAM.
But how do OS do this if there is no swap.
Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.
If there is no swap, each dirty page is locked in RAM until it can be made clean. For pages that cannot be made clean (such an anonymous or private pages), they are locked in RAM until they are no longer used.
This causes terrible performance if there are a large number of dirty pages that are not being accessed. Because they cannot be moved to swap, they directly reduce the available RAM for clean pages, which can result in an excessive amount of needless paging.