0

I'm trying to get to the bottom of an issue where the oom-killer is killing certain jobs off on a server. The server has 48GB of physical ram, with 20GB of swap, and I'm trying to find out what processes are the ones using memory. Output from free ties up with what I'd expect to see on this server, but output from both top and ps show over 600% of memory committed to processes, with a total resident memory footprint of 258GB.

PID     USER    PR  NI  VIRT    RES     SHR     S   CPU %MEM    TIME+   COMMAND
52788   usradm  20  0   47.123g 0.018   0.013t  S   0   39.35   36:06.0 tsk_W59
52793   usradm  20  0   41.668g 0.01    0.010t  S   0   22.28   44:56.8 tsk_W64
52815   usradm  20  0   41.638g 9.689   9.534g  S   0   20.49   10:34.7 tsk_W85
52813   usradm  20  0   41.665g 9.074   8.890g  S   0   19.19   48:03.6 tsk_W83
52808   usradm  20  0   41.630g 8.641   8.507g  S   0   18.28   18:32.1 tsk_W79
23445   usradm  20  0   41.661g 8.536   8.362g  S   0   18.05   02:37.8 tsk_W84
52811   usradm  20  0   41.641g 8.324   8.178g  S   2   17.6    49:12.3 tsk_W81
52769   usradm  20  0   41.604g 8.241   8.133g  S   0   17.43   22:55.0 tsk_W40
52806   usradm  20  0   41.645g 8.122   7.968g  S   0   17.18   23:23.4 tsk_W77
52812   usradm  20  0   41.622g 8.085   7.955g  S   0   17.1    45:00.1 tsk_W82
39198   usradm  20  0   41.653g 7.979   7.816g  S   0   16.87   26:59.3 tsk_W76
52754   usradm  20  0   41.601g 7.486   7.383g  S   0   15.83   16:36.8 tsk_W25
52798   usradm  20  0   41.642g 7.476   7.329g  S   0   15.81   26:48.0 tsk_W69
52755   usradm  20  0   41.604g 7.471   7.367g  S   0   15.8    12:01.3 tsk_W26
52800   usradm  20  0   41.610g 7.449   7.321g  S   0   15.75   52:18.5 tsk_W71
52799   usradm  20  0   41.625g 7.44    7.310g  S   0   15.73   27:58.2 tsk_W70
36047   usradm  20  0   41.746g 7.292   7.033g  S   0   15.42   16:20.4 tsk_W75
52801   usradm  20  0   41.633g 7.2     7.062g  S   0   15.23   27:59.8 tsk_W72
52807   usradm  20  0   41.676g 7.115   6.924g  S   1   15.05   53:07.0 tsk_W78
52803   usradm  20  0   41.641g 6.814   6.669g  S   0   14.41   27:38.2 tsk_W74
52748   usradm  20  0   41.609g 6.811   6.706g  S   0   14.41   08:55.4 tsk_W19
52744   usradm  20  0   41.602g 6.634   6.532g  S   0   14.03   06:05.1 tsk_W15
52753   usradm  20  0   41.600g 6.15    6.050g  S   0   13.01   12:39.0 tsk_W24
52750   usradm  20  0   41.602g 5.904   5.806g  S   0   12.49   09:00.7 tsk_W21
52751   usradm  20  0   41.605g 5.771   5.659g  S   0   12.2    05:46.6 tsk_W22
52745   usradm  20  0   41.605g 5.766   5.667g  S   0   12.19   06:33.6 tsk_W16
52746   usradm  20  0   41.608g 5.464   5.357g  S   0   11.56   07:31.6 tsk_W17
52752   usradm  20  0   41.614g 5.308   5.205g  S   0   11.23   09:13.3 tsk_W23
52747   usradm  20  0   41.608g 5.235   5.136g  S   0   11.07   15:23.0 tsk_W18
52749   usradm  20  0   41.598g 4.859   4.758g  S   0   10.28   07:40.8 tsk_W20
52738   usradm  20  0   41.595g 4.563   4.471g  S   0   9.651   04:21.7 tsk_W9
52742   usradm  20  0   41.599g 3.968   3.872g  S   0   8.392   03:31.4 tsk_W13
52731   usradm  20  0   41.590g 3.867   3.779g  S   0   8.178   02:07.8 tsk_W2
52741   usradm  20  0   41.595g 3.834   3.742g  S   0   8.109   05:35.7 tsk_W12
52735   usradm  20  0   41.593g 3.719   3.630g  S   0   7.866   03:10.4 tsk_W6
52737   usradm  20  0   41.598g 3.592   3.500g  S   0   7.597   05:33.7 tsk_W8
52743   usradm  20  0   41.593g 3.525   3.433g  S   0   7.456   04:45.2 tsk_W14
52740   usradm  20  0   41.599g 3.442   3.342g  S   0   7.28    05:00.9 tsk_W11
52739   usradm  20  0   41.599g 3.393   3.302g  S   0   7.175   03:53.7 tsk_W10
52736   usradm  20  0   41.597g 3.116   3.022g  S   0   6.59    03:43.0 tsk_W7
52732   usradm  20  0   41.601g 3.096   3.009g  S   0   6.547   02:35.4 tsk_W3
52734   usradm  20  0   41.596g 2.683   2.593g  S   0   5.674   02:29.6 tsk_W5
52733   usradm  20  0   41.593g 2.658   2.568g  S   0   5.622   02:35.5 tsk_W4
52729   usradm  20  0   41.598g 2.264   2.176g  S   0   4.787   04:23.5 tsk_W0
52787   usradm  20  0   41.641g 2.123   1.969g  S   0   4.491   15:26.6 tsk_W58
52730   usradm  20  0   41.597g 1.602   1.523g  S   0   3.389   02:11.1 tsk_W1
70422   usradm  20  0   41.716g 1.434   1.191g  S   0   3.034   23:37.4 tsk_W80
52802   usradm  20  0   41.656g 1.369   1.210g  S   0   2.895   54:25.8 tsk_W73
52773   usradm  20  0   41.664g 1.323   1.151g  S   0   2.799   12:22.8 tsk_W44
52782   usradm  20  0   41.633g 1.32    1.180g  S   0   2.791   29:15.6 tsk_W53
52784   usradm  20  0   41.688g 1.286   1.101g  S   0   2.72    09:17.5 tsk_W55

Can someone elaborate on how to pick this apart? I've not had a situation before where memory appears to be over provisioned on a server like this.

TIA

3
  • 1
    I'm unfamiliar with your "ps aux" output. What's the OS and release? Are you sure these aren't threads of the same process or something like this? What's "htop" output like?
    – wazoox
    Jun 8, 2022 at 16:06
  • edit your question and add the results of both pstree and sysctl -a 2>/dev/null | grep vm.overcommit.
    – mforsetti
    Jun 9, 2022 at 8:44
  • edit your question and add the results of both pstree and sysctl -a 2>/dev/null | grep vm.overcommit.
    – mforsetti
    Jun 9, 2022 at 8:44

0

You must log in to answer this question.

Browse other questions tagged .