Looks to be explained in the Nexus docs here https://help.sonatype.com/repomanager3/system-requirements#SystemRequirements-AdequateFileHandleLimits
On most Linux systems, persistent limits can be set for a particular user by editing the /etc/security/limits.conf file. To set the maximum number of open files for both soft and hard limits for the nexus user to 65536, add the following line to the /etc/security/limits.conf file, where "nexus" should be replaced with the user ID that is being used to run the repository manager:
nexus - nofile 65536
This change will only take effect the next time the nexus process user opens a new session. Which essentially means that you will need to restart NXRM.
On Ubuntu systems there is a caveat: Ubuntu ignores the /etc/security/limits.conf file for processes started by init.d.
So if NXRM is started using init.d there, edit /etc/pam.d/common-session and uncomment the following line ( remove the hash # and space at the beginning of the line):
# session required pam_limits.so
For more information refer to your specific operating system documentation.
If you're using systemd to launch the server the above won't work. Instead, modify the configuration file to add a LimitNOFILE line: