0

I am having a log file of 25 million lines. I deleted 5 million lines from that file manually using vim. Below is the disk space usage of server before the deletion of logs. I deleted nginx logs from /var/log/nginx/ folder.

$df -kh
Filesystem      Size  Used Avail Use% Mounted on
udev             15G   12K   15G   1% /dev
tmpfs           3.0G  500K  3.0G   1% /run
/dev/xvda1       48G   35G   11G  77% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             15G     0   15G   0% /run/shm
none            100M     0  100M   0% /run/user

After deletion of logs the disk space usage has gone up by almost 11 % I don't understand what has gone wrong here. Below is the disk usage after the deletion of logs.

$ df -kh
Filesystem      Size  Used Avail Use% Mounted on
udev             15G   12K   15G   1% /dev
tmpfs           3.0G  504K  3.0G   1% /run
/dev/xvda1       48G   40G  5.6G  88% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             15G     0   15G   0% /run/shm
none            100M     0  100M   0% /run/user

Any expert advice would be a great help here. Thanks

Below is output of ls -alht /var/log/nginx/

total 19G
-rw-r--r--  1 root     root   5.5G Jan  5 23:07 example-afew.access.log.1
-rw-r--r--  1 root     root   951M Jan  5 23:07 example-accountant.access.log.1
-rw-r--r--  1 root     root   1.2G Jan  5 23:07 example-outlet.access.log.1
-rw-r--r--  1 root     root   2.1G Jan  5 23:07 example-folks.access.log.1
-rw-r--r--  1 root     root   1.9G Jan  5 23:07 example-api_clients.access.log.1
-rw-r--r--  1 root     root   1.4G Jan  5 23:07 example-registry.access.log.1
-rw-r--r--  1 root     root    19M Jan  5 23:07 example-services.access.log.1
-rw-r--r--  1 root     root    54M Jan  5 23:07 example-postman-jack.access.log.1
-rw-r--r--  1 root     root   123M Jan  5 23:07 example-referrific.access.log.1
-rw-r--r--  1 root     root   167M Jan  5 23:07 example-creds.access.log.1
-rw-r--r--  1 root     root    34M Jan  5 23:06 example-blog-service.access.log.1
-rw-r--r--  1 root     root   140M Jan  5 23:06 example-help-service.access.log.1
-rw-r--r--  1 root     root    28M Jan  5 23:04 example-mqttproxy.access.log.1
-rw-r--r--  1 root     root    25M Jan  5 23:02 example-push-notifications.access.log.1
-rw-r--r--  1 root     root    12M Jan  5 22:58 example-network-coverage.access.log.1
-rw-r--r--  1 root     root   2.4M Jan  5 22:53 example-dcl_shipments.access.log.1
drwxr-xr-x  2 www-data adm    4.0K Jan  5 21:51 .
-rw-r--r--  1 root     root   5.1G Jan  5 20:13 example-subs.access.log.1
-rw-r--r--  1 root     root   9.5M Jan  5 15:16 example-screw.access.log.1
drwxrwxr-x 12 root     syslog 4.0K Jan  5 06:39 ..
-rw-r--r--  1 root     root   149K Jan  4 08:30 example-nobono.access.log.1
-rw-r-----  1 www-data adm       0 Dec 23 06:49 error.log
-rw-r-----  1 www-data adm    2.1K Dec 22 20:38 error.log.1
-rw-r-----  1 www-data adm     132 Dec 21 16:30 error.log.2.gz
-rw-r-----  1 www-data adm     155 Dec 20 14:47 error.log.3.gz
-rw-r-----  1 www-data adm       0 Jul 20 06:44 example-afew.error.log
-rw-r--r--  1 root     root   8.3K Jul 19 18:20 example-afew.error.log.1
-rw-r-----  1 www-data adm       0 Jun 24  2016 example-outlet.error.log
-rw-r--r--  1 root     root    14K Jun 23  2016 example-outlet.error.log.1
-rw-r--r--  1 root     root      0 Jun 21  2016 example-premium.access.log
-rw-r--r--  1 root     root      0 Jun 21  2016 example-premium.error.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-push-notifications.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-referrific.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-services.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-outlet.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-subs.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-afew.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-api_clients.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-blog-service.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-screw.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-nobono.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-accountant.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-dcl_shipments.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-registry.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-help-service.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-creds.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-mqttproxy.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-network-coverage.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-folks.access.log
-rw-r-----  1 www-data adm       0 Jun 19  2016 example-postman-jack.access.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-api_clients.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-blog-service.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-screw.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-nobono.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-accountant.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-dcl_shipments.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-registry.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-help-service.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-creds.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-mqttproxy.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-network-coverage.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-courier.access.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-courier.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-folks.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-postman-jack.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-push-notifications.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-referrific.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-services.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 example-subs.error.log
-rw-r--r--  1 root     root      0 Jun 17  2016 access.log
  • 1
    has vim created a backup file? – dawud Jan 5 '17 at 20:58
  • 1
    What does "of 2.5 crores of line" mean? Why would you delete log lines manually with vim? You should use log rotation. – Tero Kilkanen Jan 5 '17 at 21:01
  • @dawud where would it create a backup file? and by what name? – Shailesh Sutar Jan 5 '17 at 21:01
  • Depends on your vim configuration, but normally (if enabled) in the same location of the edited file, with a tilde extension (e.g. some_file.txt~) – dawud Jan 5 '17 at 21:03
  • 2
    "Lakh" and "crore" are virtually unknown outside India. On worldwide sites it's best to avoid these words. – Michael Hampton Jan 5 '17 at 21:54
3

Vim has write the remaining content of your log to a new inode. The old file is still open and with all previous content by the service logging in it.

Check for open deleted files with :

lsof | grep deleted

To clean disk space you must close the deleted file, for example by restarting the service.

Next time do like logrotate copytruncate : copy what you want to keep and truncate old file.

  • lsof | grep deleted shows nothing. Will a nginx reload work in order to normalize everthing back? – Shailesh Sutar Jan 6 '17 at 12:10
  • Did you tried ncdu? – Vikelidis Kostas Jan 8 '17 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.