0

We are running load test against an application that hits a Postgres database.

During the test, we suddenly get an increase in error rate. After analysing the platform and application behaviour, we notice that:

  • CPU of Postgres RDS is 100%
  • Freeable memory drops on this same server

And in the postgres logs, we see:

2018-08-21 08:19:48 UTC::@:[XXXXX]:LOG: server process (PID XXXX) was terminated by signal 9: Killed

After investigating and reading documentation, it appears one possibility is linux oomkiller running having killed the process.

But since we're on RDS, we cannot access system logs /var/log messages to confirm.

So can somebody:

  • confirm that oom killer really runs on AWS RDS for Postgres
  • give us a way to check this ?
  • give us a way to compute max memory used by Postgres based on number of connections ?

I didn't find the answer here:

1

Even if the OOM killer did not act (it probably did), sustained 100% CPU and very low free memory is bad for performance.

Use a larger instance size and see if the problem goes away. Test a smaller size on a non-RDS Postgres you control and see if the OOM killer gets angry.

Number of connections is not necessarily the dominating factor in memory consumption: shared memory is used for other things, and not every query uses a large chunk of memory. See also this conversation: PostgreSql allocate memory for each connection.

Additional advice from Best Practices for Amazon RDS

DB Instance RAM Recommendations

An Amazon RDS performance best practice is to allocate enough RAM so that your working set resides almost completely in memory. To tell if your working set is almost all in memory, check the ReadIOPS metric (using Amazon CloudWatch) while the DB instance is under load. The value of ReadIOPS should be small and stable. If scaling up the DB instance class—to a class with more RAM—results in a dramatic drop in ReadIOPS, your working set was not almost completely in memory. Continue to scale up until ReadIOPS no longer drops dramatically after a scaling operation, or ReadIOPS is reduced to a very small amount.

Evaluating Performance Metrics

Freeable Memory – How much RAM is available on the DB instance, in megabytes. The red line in the Monitoring tab metrics is marked at 75% for CPU, Memory and Storage Metrics. If instance memory consumption frequently crosses that line, then this indicates that you should check your workload or upgrade your instance.

0

I do not have much experience with Postgres, but in the same situation I find out that an RDS MySql instance is prone to reboot entirely. Even if you do not have access to the underlying systems you should be able to get postgres logs thru the web console. Look for a reboot the daemon should indicate that it is closing and starting.

Anyway if you are working in danger zone. there's not much you can do. You should move to an instance with more RAM / CPU available.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.