2

We are using ceph version 14.2.0. We have 4 hosts with 24 BlueStore OSDs, each is 1.8TB (2TB spinning disk). We have only a single pool with size 2 and I am absolutely sure that we are using more space than what ceph df shows:

[root@blackmirror ~]# ceph osd dump | grep 'replicated size'
pool 2 'one' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 900 pgp_num 900 autoscale_mode warn last_change 37311 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd

[root@blackmirror ~]# ceph df
RAW STORAGE:
CLASS     SIZE       AVAIL      USED       RAW USED     %RAW USED
    hdd       44 TiB     21 TiB     22 TiB       23 TiB         51.61
    TOTAL     44 TiB     21 TiB     22 TiB       23 TiB         51.61

POOLS:
    POOL     ID     STORED      OBJECTS     USED        %USED     MAX AVAIL
    one       2     2.7 TiB       2.94M     5.5 TiB     28.81       6.7 TiB

Not sure about MAX AVAIL, but I think it's wrong too.

Here's the output of ceph osd df:

ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE  DATA     OMAP    META    AVAIL   %USE  VAR  PGS STATUS
 0   hdd 1.81310  1.00000 1.8 TiB  1.2 TiB  1.2 TiB 152 KiB 2.8 GiB 669 GiB 64.10 1.24  94     up
 1   hdd 1.81310  1.00000 1.8 TiB  937 GiB  935 GiB  80 KiB 2.2 GiB 926 GiB 50.31 0.97  72     up
 2   hdd 1.81310  1.00000 1.8 TiB  788 GiB  786 GiB  36 KiB 1.9 GiB 1.0 TiB 42.33 0.82  65     up
 3   hdd 1.81310  1.00000 1.8 TiB  868 GiB  866 GiB 128 KiB 2.1 GiB 995 GiB 46.59 0.90  69     up
 4   hdd 1.81310  1.00000 1.8 TiB  958 GiB  956 GiB  84 KiB 2.3 GiB 904 GiB 51.45 1.00  72     up
 5   hdd 1.81879  1.00000 1.8 TiB 1015 GiB 1013 GiB  64 KiB 2.4 GiB 847 GiB 54.50 1.06  77     up
 6   hdd 1.81310  1.00000 1.8 TiB 1015 GiB 1012 GiB  32 KiB 2.6 GiB 848 GiB 54.48 1.06  81     up
 7   hdd 1.81310  1.00000 1.8 TiB  935 GiB  932 GiB  40 KiB 2.3 GiB 928 GiB 50.18 0.97  70     up
 8   hdd 1.81310  1.00000 1.8 TiB  1.0 TiB  1.0 TiB  48 KiB 2.5 GiB 800 GiB 57.05 1.11  83     up
 9   hdd 1.81310  1.00000 1.8 TiB 1002 GiB 1000 GiB  96 KiB 2.3 GiB 861 GiB 53.79 1.04  77     up
10   hdd 1.81310  1.00000 1.8 TiB  779 GiB  777 GiB 168 KiB 1.9 GiB 1.1 TiB 41.80 0.81  63     up
11   hdd 1.81310  1.00000 1.8 TiB  1.1 TiB  1.1 TiB 128 KiB 2.6 GiB 768 GiB 58.77 1.14  83     up
12   hdd 1.81310  1.00000 1.8 TiB  798 GiB  796 GiB 120 KiB 1.9 GiB 1.0 TiB 42.85 0.83  67     up
13   hdd 1.81310  1.00000 1.8 TiB  1.1 TiB  1.1 TiB  64 KiB 2.6 GiB 761 GiB 59.12 1.15  89     up
14   hdd 1.81310  1.00000 1.8 TiB  1.2 TiB  1.2 TiB 128 KiB 2.7 GiB 680 GiB 63.51 1.23  88     up
15   hdd 1.81310  1.00000 1.8 TiB  766 GiB  764 GiB  64 KiB 1.9 GiB 1.1 TiB 41.15 0.80  58     up
16   hdd 1.81310  1.00000 1.8 TiB  990 GiB  988 GiB  80 KiB 2.4 GiB 873 GiB 53.15 1.03  81     up
17   hdd 1.81310  1.00000 1.8 TiB  980 GiB  977 GiB  80 KiB 2.3 GiB 883 GiB 52.61 1.02  77     up
18   hdd 1.81310  1.00000 1.8 TiB  891 GiB  890 GiB  68 KiB 1.7 GiB 971 GiB 47.87 0.93  73     up
19   hdd 1.81310  1.00000 1.8 TiB  1.1 TiB  1.1 TiB  60 KiB 2.0 GiB 784 GiB 57.87 1.12  87     up
20   hdd 1.81310  1.00000 1.8 TiB  956 GiB  955 GiB  48 KiB 1.8 GiB 906 GiB 51.37 1.00  73     up
21   hdd 1.81310  1.00000 1.8 TiB  762 GiB  760 GiB  32 KiB 1.6 GiB 1.1 TiB 40.91 0.79  58     up
22   hdd 1.81310  1.00000 1.8 TiB  979 GiB  977 GiB  80 KiB 1.9 GiB 883 GiB 52.60 1.02  72     up
23   hdd 1.81310  1.00000 1.8 TiB  935 GiB  934 GiB 164 KiB 1.8 GiB 927 GiB 50.24 0.97  71     up
                    TOTAL  44 TiB   23 TiB   22 TiB 2.0 MiB  53 GiB  21 TiB 51.61
MIN/MAX VAR: 0.79/1.24  STDDEV: 6.54

And here is the output of rados df

[root@blackmirror ~]# rados df
POOL_NAME    USED OBJECTS CLONES  COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED      RD_OPS      RD      WR_OPS      WR USED COMPR UNDER COMPR
one       5.5 TiB 2943372      0 5886744                  0       0        0 11291297816 114 TiB 24110141554 778 TiB        0 B         0 B

total_objects    2943372
total_used       23 TiB
total_avail      21 TiB
total_space      44 TiB

In reality we are storing around 11TB of data, so total_used above looks right because our replication size is 2.

This started happening after we changed OSDs 18-23. They were initially 1TB disks, but we upgraded them to 2TB to balance the cluster. After we changed the first disk, USED and MAX AVAIL from ceph df dropped to around 1TB. I thought this is just a matter of time, but even after all recovery operations has finished, we are left with the picture above. I have tried to force a deep scrub on all disks, which nearly killed all applications in the cluster for 12 hours, but it did nothing at the end. I am clueless as to what to do now. Please help.

0

You must log in to answer this question.

Browse other questions tagged .