1

I have a default Solaris 10 installation running on a 64GB box. For some reason, not all of the memory is available for use. The reason I'm even bothered by it, is because I'm trying to get mysql to run there, and give it 58GB of RAM, or more, but the maximum I can map is 45GB (nmap fails).

Nothing is running, except the GUI (which we will remove soon). But the UI doesn't explain the fact that 13 GB of the memory are gone, somewhere.

If ARC is a part of the used memory in Solaris (as opposed to the filecache in linux), then the UI consumes 5GB, and we still should have at least 55GB of RAM for MySQL. BUT: if this memory is consumed by the ARC, it should give the memory up, when innodb tries to nmap the memory (or does it?), so it makes no sense.

How can I see what consumes this memory, and free it?

Some technical data:

#kstat -m zfs | grep size
    data_size                       7847447040
    hdr_size                        139495272
    l2_hdr_size                     0
    l2_size                         0
    other_size                      510852000
    size                            8497794312

# top
96 processes: 95 sleeping, 1 on cpu
CPU states: 99.9% idle,  0.0% user,  0.1% kernel,  0.0% iowait,  0.0% swap
Memory: 64G phys mem, 51G free mem, 2048M swap, 2048M free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  1529 root       1  59    0   67M   10M sleep   11:17  0.20% mixer_applet2
   902 root       1  59    0   12M 9848K sleep    4:33  0.08% gconfd-2
  1923 root      34  59    0  165M   57M sleep    3:50  0.06% java
  1525 root       1  59    0   69M   13M sleep    3:24  0.06% gnome-netstatus
   644 root       1  58    0   45M   29M sleep    2:38  0.05% Xorg
  1501 root       1  59    0   73M   18M sleep    2:16  0.04% gnome-panel
 19814 root       1  59    0 3268K 1820K cpu      0:00  0.03% top
  2416 root      47  59    0   87M   68M sleep    2:08  0.03% java
 22587 noaccess  18  59    0  131M  115M sleep    2:13  0.03% java
   744 noaccess  18  59    0  132M  115M sleep    2:13  0.03% java
   908 root       1  59    0 6608K 4020K sleep    0:09  0.01% xscreensaver
  2435 root       2  49    0   73M   14M sleep    0:18  0.01% gnome-terminal
  2497 root      23  59    0   11M 7384K sleep    0:04  0.01% nscd
 22367 root       1  59    0 9832K 6636K sleep    0:01  0.00% snmpd
 15253 root       1  59    0 6516K 3840K sleep    0:00  0.00% sshd

# prtdiag -v
System Configuration: Dell Inc. PowerEdge R710
BIOS Configuration: Dell Inc. 2.2.10 11/09/2010
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz CPU1
Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz CPU2

==== Memory Device Sockets ================================

Type    Status Set Device Locator      Bank Locator
------- ------ --- ------------------- --------------------
Unknown in use 1   DIMM_A1             
Unknown in use 1   DIMM_A2             
Unknown empty  2   DIMM_A3             
Unknown in use 2   DIMM_A4             
Unknown in use 3   DIMM_A5             
Unknown empty  3   DIMM_A6             
Unknown empty  4   DIMM_A7             
Unknown empty  4   DIMM_A8             
Unknown empty  5   DIMM_A9             
Unknown in use 5   DIMM_B1             
Unknown in use 6   DIMM_B2             
Unknown empty  6   DIMM_B3             
Unknown in use 4   DIMM_B4             
Unknown in use 5   DIMM_B5             
Unknown empty  6   DIMM_B6             
Unknown empty  4   DIMM_B7             
Unknown empty  5   DIMM_B8             
Unknown empty  6   DIMM_B9             

==== On-Board Devices =====================================
Embedded Matrox G200 Video
Embedded Broadcom 5709C NIC 1
Embedded Broadcom 5709C NIC 2
Embedded Broadcom 5709C NIC 3
Embedded Broadcom 5709C NIC 4
Integrated RAID Controller

==== Upgradeable Slots ====================================

ID  Status    Type             Description
--- --------- ---------------- ----------------------------
1   available Unknown          PCI1
2   available Unknown          PCI2
3   in use    Unknown          PCI3
4   available Unknown          PCI4

#prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 21369 root     3888K 3036K cpu10   59    0   0:00:00 0.0% prstat/1
  1529 root       67M   10M sleep   59    0   0:11:22 0.0% mixer_applet2/1
 21234 100        45G 3686M sleep   59    0   0:00:17 0.0% mysqld/25
   902 root       12M 9848K sleep   59    0   0:04:35 0.0% gconfd-2/1
   644 root       45M   29M sleep   59    0   0:02:39 0.0% Xorg/1
   642 root     2560K  764K sleep   59    0   0:00:00 0.0% fbconsole/1
   640 root     5704K 1408K sleep   59    0   0:00:00 0.0% dtlogin/1
   863 root     4872K 1968K sleep   59    0   0:00:00 0.0% sdt_shell/1
  2438 root     1432K  832K sleep   59    0   0:00:00 0.0% sh/1
  2436 root     2644K 1392K sleep   59    0   0:00:00 0.0% gnome-pty-helpe/1
   883 root     1532K  892K sleep   59    0   0:00:00 0.0% Xsession2.jds/1
  2435 root       73M   14M sleep   49    0   0:00:18 0.0% gnome-terminal/2
   789 root     1568K  940K sleep   59    0   0:00:00 0.0% Xsession/1
   783 smmsp    9452K 2120K sleep   59    0   0:00:00 0.0% sendmail/1
   864 root     3228K  836K sleep   59    0   0:00:00 0.0% dsdm/1
   435 root     2064K  940K sleep   59    0   0:00:00 0.0% smcboot/1
   436 root     2064K  684K sleep   59    0   0:00:00 0.0% smcboot/1
   402 root     1432K  624K sleep   59    0   0:00:00 0.0% utmpd/1
   866 root     1456K  840K sleep   59    0   0:00:00 0.0% sh/1
   437 root     2064K  684K sleep   59    0   0:00:00 0.0% smcboot/1
   407 root     3988K 1936K sleep   59    0   0:00:00 0.0% syslogd/11
   630 root     3256K 2260K sleep   59    0   0:00:00 0.0% vold/5
  2497 root       11M 7392K sleep   59    0   0:00:04 0.0% nscd/23
 22200 root     3876K 1568K sleep   59    0   0:00:00 0.0% syslogd/13
   399 root     7996K 3996K sleep   59    0   0:00:01 0.0% inetd/4
   415 root     2468K 1372K sleep   59    0   0:00:00 0.0% ttymon/1
   404 root     2504K 1276K sleep   59    0   0:00:00 0.0% ttymon/1
   295 daemon   2932K 1432K sleep   59    0   0:00:00 0.0% rpcbind/1
   142 root     5452K 2184K sleep   59    0   0:00:00 0.0% syseventd/15
   224 root     2824K 1080K sleep   59    0   0:00:00 0.0% cron/1
   156 root     4344K 3164K sleep   59    0   0:00:00 0.0% picld/9
   387 root       18M   14M sleep   59    0   0:00:02 0.0% fmd/19
   152 daemon   4612K 2684K sleep   59    0   0:00:00 0.0% kcfd/4
   397 root     2132K 1040K sleep   59    0   0:00:00 0.0% sac/1
 19861 root     5864K 3720K sleep   59    0   0:00:00 0.0% zlogin/1
   204 root     2244K  780K sleep   59    0   0:00:00 0.0% iscsi-initiator/2
  2498 daemon   2716K 1676K sleep   59    0   0:00:00 0.0% statd/1
   369 root     2528K 1292K sleep   59    0   0:00:03 0.0% in.routed/1
    77 root     6148K 3484K sleep   59    0   0:00:02 0.0% devfsadm/9
  1923 root      165M   57M sleep   59    0   0:03:51 0.0% java/34
   669 root     6600K 2912K sleep   59    0   0:00:00 0.0% dtlogin/1
    11 root       12M   11M sleep   59    0   0:00:08 0.0% svc.configd/16
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    91 root      423M  425M   0.6%   0:34:57 0.0%
     1 100        45G 3703M   5.7%   0:00:17 0.0%
     2 noaccess  251M  237M   0.4%   0:04:28 0.0%
     2 smmsp    3804K   16M   0.0%   0:00:00 0.0%
    10 daemon     10M   14M   0.0%   0:00:00 0.0%

#  echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     646470              2525    4%
ZFS File Data             1920208              7500   11%
Anon                      1078294              4212    6%
Exec and libs               14431                56    0%
Page cache                   2544                 9    0%
Free (cachelist)            13012                50    0%
Free (freelist)          13096905             51159   78%

Total                    16771864             65515
Physical                 16324293             63766
Improve this question

3 Answers 3

Reset to default
3

Can please paste a out of following

prtdiag -v
prstat -a
Improve this answer
5
  • Gladly. Added to the question.
    – shlomoid
    May 2, 2011 at 10:11
  • 1
    # echo ::memstat | mdb -k one more out put please
    – Rajat
    May 2, 2011 at 10:23
  • Added. And this last excellent command actually explains where it's gone!
    – shlomoid
    May 2, 2011 at 10:31
  • What does "Anon" stand for? And why is the kernel 2.5GB in size?
    – shlomoid
    May 2, 2011 at 10:32
  • From the app management guide: "Anonymous memory means the memory pages that are not associated with files. For example, anonymous memory is used for process heaps and stacks"
    – gtirloni
    May 2, 2011 at 12:08
4

Based on the ::memstat output, ZFS is using up 7.5GB of physical memory. Be careful with ZFS: it can often be too aggressive and will starve the rest of the system of memory. In practice it simply doesn't release memory when you'd like it to.

See the ZFS Best Practices Guide for details on how you can limit the ARC size to mitigate this issue.

Improve this answer
0

Is there any project limit in the memory mysql can allocate?

prctl -i project default

or, instead of 'default', the project under which mysql runs?

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .