121 reputation
5
bio website
location
age
visits member for 2 years, 11 months
seen May 18 at 18:51

Software engineer with working experience in Java, Python, Ruby, AWK and associated technologies.


Dec
9
comment Why can't I malloc as much memory as `free` reports as being free?
I was waiting for Michael Hampton to post an answer, so I could award him the accepted answer. I have more often added a short version of the accepted answer at the top of a post, to spare people the trouble of wading through often long questions and answers to determine whether they are running into the same problem. After the answer is clear, it often turns out the question could have been much shorter. I think a short tl;dr at the top can be very helpful, even if a more complete answer is accepted.
Dec
9
awarded  Informed
Dec
8
revised Why can't I malloc as much memory as `free` reports as being free?
added 179 characters in body
Dec
8
awarded  Commentator
Dec
8
comment Why can't I malloc as much memory as `free` reports as being free?
Ah, fragmentation, I hadn't thought of that. That makes sense, thanks. I've now been looking at the OOM output more closely, but I don't know how to identify the fragmentation, so if you could explain that in your answer: awesome. If there's a way to reduce the fragmentation by smarter use of malloc (or perhaps a library that wraps malloc): even better.
Dec
8
comment Why can't I malloc as much memory as `free` reports as being free?
Added OOM killer output. What do you expect to see there? I don't see anything out of the ordinary.
Dec
8
revised Why can't I malloc as much memory as `free` reports as being free?
added 23461 characters in body
Dec
8
asked Why can't I malloc as much memory as `free` reports as being free?
Dec
6
revised How can used physical memory exceed sum of 'resident' memory reported by processes?
added 746 characters in body
Dec
6
asked How can used physical memory exceed sum of 'resident' memory reported by processes?
Dec
5
comment How can free physical memory decrease while 'Res' memory used by processes is constant?
Can GC cycles explain hundreds of MB's of pages slowly being written to over a period of several hours? As far as I understand GC, the effect would happen pretty quickly after operations were suspended? Can it also explain the sudden increase in the rate of increase and the decrease that can be noticed?
Dec
5
comment How can free physical memory decrease while 'Res' memory used by processes is constant?
Let's assume the observations are explained by 'pages being copied lazily'. Then my followup question is: exactly what Linux or Ruby operation can cause hundreds of MB's of pages to be copied lazily over a period of several hours, while these processes aren't using any CPU and are not processing any input or performing any cron-like jobs? My goal is to understand what is happening to such an extent that I can intervene and prevent this from happening.
Dec
5
comment How can free physical memory decrease while 'Res' memory used by processes is constant?
That explains why the sum of 'RES' can be much larger than the amount of physically available memory, which I already acknowledge in my second footnote. It doesn't answer the question I am asking: how can RES stay the same, while actually used physical memory increases, even though not much is happening on the machine (as evidenced by the load)?
Dec
5
revised How can free physical memory decrease while 'Res' memory used by processes is constant?
added 269 characters in body
Dec
5
comment How can free physical memory decrease while 'Res' memory used by processes is constant?
The amount of shared memory reported by top as SHR is always quite low, ~4000 (kB) and doesn't change significantly.
Dec
5
comment How can free physical memory decrease while 'Res' memory used by processes is constant?
I added a graph of our NewRelic monitoring during that time. I was closely watching the machine during the day, which I why I have enough data points to know for sure their data exactly matches what top reports. I don't have any captures of top itself and recreating those would be a pain, which I don't think adds much.
Dec
5
revised How can free physical memory decrease while 'Res' memory used by processes is constant?
added 321 characters in body
Dec
5
revised How can free physical memory decrease while 'Res' memory used by processes is constant?
edited title
Dec
5
awarded  Editor
Dec
5
revised How can free physical memory decrease while 'Res' memory used by processes is constant?
added 2 characters in body