I am getting stuck with an issue for a couple of days, and hope someone here could help or give some hints.
So here is the issue: when I am running some docker images which performs large I/O operations, the container become read-only in the middle of the run. My docker container is running in AWS ECS on AWS EC2. The image I am running is based on here https://hub.docker.com/r/jenkinsci/jnlp-slave/.
I have updated the docker base storage size to 200G and when the container is running I can see it definitely has enough space for the file that it's generating so it doesn't look like an disk space issue. I also did some google search and it seems like the host machine is encountering XFS ENOSPC error - but I have no idea what this means and how to resolve it.
When I check the dmesg logs in the host machine EC2 instance I can see below errors:
[Tue Oct 16 06:16:14 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode
[Tue Oct 16 06:17:15 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
[Tue Oct 16 06:17:15 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5588992)
[Tue Oct 16 06:17:15 2018] buffer_io_error: 585671 callbacks suppressed
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588992
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588993
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588994
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588995
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588996
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588997
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588998
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588999
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589000
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589001
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5589504)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590016)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590528)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591040)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591552)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592064)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592576)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593088)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593600)
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] Aborting journal on device dm-4-8.
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs (dm-4): Remounting filesystem read-only
Below is the EC2 spec:
OS: Linux ip-10-119-5-115 4.14.59-64.43.amzn1.x86_64 #1 SMP Thu Aug 2 21:29:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker info:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 18.06.1-ce
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 214.7GB
Backing Filesystem: ext4
Udev Sync Supported: true
Data Space Used: 802.2MB
Data Space Total: 23.33GB
Data Space Available: 22.53GB
Metadata Space Used: 606.2kB
Metadata Space Total: 25.17MB
Metadata Space Available: 24.56MB
Thin Pool Minimum Free Space: 2.333GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.59-64.43.amzn1.x86_64
Operating System: Amazon Linux AMI 2018.03
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.1GiB
Name: ip-10-119-5-115
ID: NJ4J:RBYM:3A25:QOZJ:HRSH:6BPD:3UO3:FW3X:4ZMI:5NWV:6IHV:OOHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 58
Goroutines: 86
System Time: 2018-10-16T06:19:26.381414812Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Built: Tue Oct 2 18:06:45 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a/18.06.1-ce
Built: Tue Oct 2 18:08:26 2018
OS/Arch: linux/amd64
Experimental: false
