2

How do you define the Ceph OSD Disk Partition Size?

It always creates with only 10 GB usable space.

  • Disk size = 3.9 TB
  • Partition size = 3.7 TB
  • Using ceph-disk prepare and ceph-disk activate (See below)
  • OSD created but only with 10 GB, not 3.7 TB

.

Commands Used

root@proxmox:~# ceph-disk prepare --cluster ceph --cluster-uuid fea02667-f17d-44fd-a4c2-a8e19d05ed51 --fs-type xfs /dev/sda4

meta-data=/dev/sda4              isize=2048   agcount=4, agsize=249036799 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=996147194, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=486399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

.

root@proxmox:~# ceph-disk activate /dev/sda4

creating /var/lib/ceph/tmp/mnt.jyqJTM/keyring
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQAohgpdjwb3NRAAIrINUiXDWQ5iMWp4Ueah3Q== with 0 caps)
got monmap epoch 3
2019-06-19 19:59:54.006226 7f966e628e00 -1 bluestore(/var/lib/ceph/tmp/mnt.jyqJTM/block) _read_bdev_label failed to open /var/lib/ceph/tmp/mnt.jyqJTM/block: (2) No such file or directory
2019-06-19 19:59:54.006285 7f966e628e00 -1 bluestore(/var/lib/ceph/tmp/mnt.jyqJTM/block) _read_bdev_label failed to open /var/lib/ceph/tmp/mnt.jyqJTM/block: (2) No such file or directory
2019-06-19 19:59:55.668619 7f966e628e00 -1 created object store /var/lib/ceph/tmp/mnt.jyqJTM for osd.0 fsid fea02667-f17d-44fd-a4c2-a8e19d05ed51
Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /lib/systemd/system/ceph-osd@.service.

# Don't worry about my keys/IDs, its just a dev environment.

.

Disk Layout

root@proxmox:~# fdisk -l
Disk /dev/sda: 3.9 TiB, 4294967296000 bytes, 8388608000 sectors
OMITTIED

Device         Start        End    Sectors   Size Type
/dev/sda1         34       2047       2014  1007K BIOS boot
/dev/sda2       2048    1050623    1048576   512M EFI System
/dev/sda3    1050624  419430400  418379777 199.5G Linux LVM
/dev/sda4  419430408 8388607966 7969177559   3.7T Ceph OSD

.

Ceph OSD Disk Size Incorrect (10 GB not 1.7 TB)

root@proxmox:~# ceph status
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0B
    usage:   1.00GiB used, 9.00GiB / 10GiB avail
    pgs:     

.


.

Full Install Details

If you want details on Proxmox install and creating Ceph OSD with partitions, read on...

Setup

  • Disk Size: 2TB NVMe (/dev/sda)
  • Operating system (Proxmox) installed with 200 GB and rest of disk is empty (1800 GB).
  • Once booted and in the web interface, create a cluster and join two hosts to ensure a green quorum status
  • Now do script below

Config Script

# Install Ceph
pveceph install

# Configure Network (Just run on Primary Proxmox Server, your LAN network)
pveceph init --network 192.168.6.0/24

# Create Monitor
pveceph createmon

# View Disks Before
sgdisk --print /dev/sda

sgdisk --largest-new=4 --change-name="4:CephOSD" \
--partition-guid=4:4fbd7e29-9d25-41b8-afd0-062c0ceff05d \
--typecode=4:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sda
  
# View Disks After (Compare)
sgdisk --print /dev/sda

# Reboot for changes to be in affect
reboot

# Note your cluster ID (fsid) at this point from the web interface 
Datacenter > Server > Ceph

# Prepare the Ceph OSD Disk, replace cluster-uuid with above fsid
ceph-disk prepare --cluster ceph --cluster-uuid fea02667-f17d-44fd-a4c2-a8e19d05ed51 --fs-type xfs /dev/sda4

# Activate the Ceph OSD Disk 
ceph-disk activate /dev/sda4

# Check Ceph OSD Disk Size
ceph status

Notes I have read posts highly recommending to use disks instead of partitions because of performance issues, I understand the warnings, but In my case I'm using NVMe SSD storage and accept any risks.

1
  • Never found an answer. Had to pay nearly double for servers with additional physical drives, Agh! – rwcommand Jun 10 '20 at 15:14
0

We ran into a similar issue recently where our OSD was limited to only 10GB despite a much larger actual disk space and found that it's related to the bluestore backend in Ceph. When bluestore is backed by a file, if that file does not already exist, it will create it:

bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory
bluestore(/var/lib/ceph/osd/ceph-0) mkfs generated fsid db688b70-0f63-4086-ba4a-ef36dc4a3147

and then resize it based on the bluestore_block_size configuration:

bluestore(/var/lib/ceph/osd/ceph-0) _setup_block_symlink_or_file resized block file to 10 GiB

The default value for bluestore_block_size in ceph version 14.2.2, for example, is 10GB [1]:

    Option("bluestore_block_size", Option::TYPE_SIZE, Option::LEVEL_DEV)
    .set_default(10_G)
    .set_flag(Option::FLAG_CREATE)
    .set_description("Size of file to create for backing bluestore"),

This can be adjusted by setting the following in your ceph.conf:

[global]
bluestore_block_size = <block file size in bytes>

[1] https://github.com/ceph/ceph/blob/v14.2.2/src/common/options.cc#L4338-L4341

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.