I am using Ubuntu Xenial, zfs and PostgreSQL 9.5 with all updates available atm. All testing is done on EC2 m4.xlarge (8GB RAM) and table size ~4GB. PostgreSQL effective_cache_size=6GB.
First scenarion: primarycache=metadata, arc_max_size=1GB (tested 4 and 6GB too), arc_size=~160mb, but Ubuntu does not use buff/cache for sequential scans (free -m says buff/cache=200MB and does not grow) and instead reads data from the disk every time query is executed. So queries are very slow.
Second scenario: primarycache=all, arc_max_size=4GB. Sequential reads are cached, but cache becomes invalidated (I see that Postgres reads data from the disk) whenever database is changed (Postgres receives updates from the master).
What I am doing wrong?
cat /etc/modprobe.d/zfs.conf options zfs zfs_prefetch_disable=1 options zfs zfs_nocacheflush=1 options zfs zfs_arc_max=7073741824