1

I have a ceph cluster with 2 nodes and 3 osd's each. Each osd is on one partion on a 8TB disk. The server is limited in amount of disk so I even have system on the same disks.

On install I used prepare and activate on the partion and everything got started up fine.

ceph-deploy osd prepare burk11:/dev/sda2 burk11:/dev/sdb2 burk11:/dev/sdd2
ceph-deploy osd activate burk12:/dev/sda2 burk12:/dev/sdb2 burk12:/dev/sdd2

It works fine. But after little testing I was planing of use it, start copied data to it and it got full very quickly and when looking at the size its only 60GB and I can't figure out why.

ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    61440M     19593M       41846M         68.11 
POOLS:
    NAME                          ID     USED       %USED     MAX AVAIL     OBJECTS 
    cinder-backup                 1           0         0         1851M           0 
    cinder-volumes                2           0         0         1851M           0 
    glance-images                 3           0         0         1851M           0 
    .rgw.root                     4        1113         0         1851M           4 
    default.rgw.control           5           0         0         1851M           8 
    default.rgw.meta              6        1661         0         1851M          10 
    default.rgw.log               7           0         0         1851M         207 
    default.rgw.buckets.index     8           0         0         1851M           2 
    default.rgw.buckets.data      9      14109M     88.40         1851M       38929 

But the disk are almost empty

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc2       7.3T  4.0G  7.3T   1% /var/lib/ceph/osd/ceph-4
/dev/sdd2       7.3T  8.4G  7.3T   1% /var/lib/ceph/osd/ceph-5
/dev/sda2       7.3T  8.2G  7.3T   1% /var/lib/ceph/osd/ceph-3


root@burk11:~# ceph osd tree
ID CLASS WEIGHT  TYPE NAME       STATUS REWEIGHT PRI-AFF 
-1       0.05878 root default                            
-3       0.02939     host burk11                         
 0   hdd 0.00980         osd.0       up  1.00000 1.00000 
 1   hdd 0.00980         osd.1       up  0.95001 1.00000 
 2   hdd 0.00980         osd.2       up  1.00000 1.00000 
-5       0.02939     host burk12                         
 3   hdd 0.00980         osd.3       up  1.00000 1.00000 
 4   hdd 0.00980         osd.4       up  1.00000 1.00000 
 5   hdd 0.00980         osd.5       up  0.95001 1.00000 
root@burk11:~# ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE  VAR  PGS TYPE NAME       
-1       0.05878        -   966G   954G 11905M 98.80 1.00   - root default    
-3       0.02939        - 30720M 21579M  9140M 70.24 0.71   -     host burk11 
 0   hdd 0.00980  1.00000 10240M  5650M  4589M 55.18 0.56 172         osd.0   
 1   hdd 0.00980  0.95001 10240M  8380M  1859M 81.84 0.83 139         osd.1   
 2   hdd 0.00980  1.00000 10240M  7548M  2691M 73.72 0.75 153         osd.2   
-5       0.02939        -   936G   933G  2764M 99.71 1.01   -     host burk12 
 3   hdd 0.00980  1.00000 10240M  9589M   650M 93.65 0.95 163         osd.3   
 4   hdd 0.00980  1.00000 10240M  9309M   930M 90.91 0.92 144         osd.4   
 5   hdd 0.00980  0.95001   916G   914G  1183M 99.87 1.01 157         osd.5   
                    TOTAL   966G   954G 11905M 98.80                          
MIN/MAX VAR: 0.56/1.01  STDDEV: 22.15

(And no, I don't use 914G on OSD.5 I have tried to expand the block file with dd, I did get the osd bigger but even if it's not using more data I got it full, should have a usage around 9-10G)

So how can I make ceph use the full partion?

  • Can you please provide output of: ceph osd tree ceph osd df tree – MaksaSila Feb 25 '18 at 18:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.