Questions tagged [ceph]

Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Erasure code overhead for heterogeneous hardware

I need to setup a Ceph cluster on a set of 10 machines having different disk sizes and numbers. The numbers ranges from 16x500GB to 8x4TB, with various intermediate values, the most common being 8x2TB....
user avatar
0 votes
0 answers
18 views

Is there a way to check filesystem/files on a logical drive in CEPH storage environment

Little background. HP C7K hardware Example partition: /dev/sdi : /dev/sdi1 ceph data, prepared, cluster ceph, osd.21, block /dev/sdi2, block.db /dev/nvme1n1p3, block.wal /dev/nvme1n1p4 /dev/sdi2 ...
user avatar
  • 1
0 votes
1 answer
91 views

Remove orphaned Ceph volumes

I have installed a Ceph cluster across three nodes which needed completely reinstalling after testing. It seems that on one of the nodes, some configuration data remained which Ceph is still picking ...
user avatar
  • 1,269
0 votes
0 answers
58 views

Usage stats of CephFS subvolumes

Is there a way to gather and show the disk usage stats of CephFS subvolumes? Is there anything similar to rbd du for CephFS subvolumes? The only idea I have is something really ugly: filesystem="...
user avatar
1 vote
0 answers
47 views

Computing stripe count for erasure coded storage

I'm setting up a ceph cluster (first time for me) which in the end will be made of ~100 disks spread over 10 hosts. I'm going with a single erasure coded data pool to maximize disk space; my ...
user avatar
0 votes
1 answer
70 views

Ceph RGW 16.2.7 CLI changes

I am trying to run a new Ceph cluster with Rados GW using the last software version 16.2.7, but when I set up RGW nodes I found out there are some changes in the CLI comparing with a version 16.2.4 I ...
user avatar
0 votes
0 answers
87 views

Openstack Glance Image Corrupted

I have an openstack instance up and running for 4 months and would like to migrate it to another nova node. When I click migrate instance option of that 4 month running instance on openstack dashboard....
user avatar
  • 33
0 votes
0 answers
80 views

Deploying Ceph with RGW Ingress uses [::]:Port instead of virtual ip

I'm deploying regular Ceph Pacific using cephadm, including RGW. Now I want to use the ingress-service, which includes specifying a virtual ip - so far so good. Unfortunately this service now creates ...
user avatar
0 votes
2 answers
1k views

ssh-copy-id ceph@osd-0 /bin/ssh-copy-id: ERROR: failed to open ID file '/home/ceph/.pub': No such file or directory

I can ssh successfully my ceph osd nodes but when i am trying to copy ssh id i am getting below error can anyone guide why i am getting below errors. [ceph@monitor ~]$ ssh-copy-id ceph@osd-0 /bin/ssh-...
user avatar
  • 11
2 votes
0 answers
674 views

Proxmox on Ceph performance & stability issues / Configuration doubts

We have just installed a cluster of 6 Proxmox servers, using 3 nodes as Ceph storage, and 3 nodes as compute nodes. We are experiencing strange and critical issues with the performances and stability ...
user avatar
  • 163
1 vote
0 answers
102 views

Ceph RGW: slow `list_bucket` requests

I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds: $ ...
user avatar
0 votes
1 answer
45 views

Changing Ceph to store number of data copies

Currently I am using replication for data placement and think I am using three copies, which I think is the default. How do I change ceph config to store 4 copies on different nodes in different ...
user avatar
0 votes
1 answer
110 views

Three Single drive Raid 1 vs JBOD

I have an intel server of which the RAID Controller does not support JBOD mode (same problem as my Dell R820s). But I want to do Ceph, so I "need" a JBOD. So I set my 3 SSDs as single drive ...
user avatar
  • 101
3 votes
0 answers
127 views

Openstack Nova and Ceph Volume Attachment Issue

I am trying out the volume attachment function at Openstack (version:wallaby) to the server as additional device but failed. The volume backend is ceph which all of the osds are up and healthy. ceph-...
user avatar
  • 33
0 votes
1 answer
42 views

Openstack Wallaby Volume Backup Issue

Recently I am testing kubernetes with 1 master node and 3 worker nodes that built on Openstack(version: Wallaby). It involves lots of testing and therefore I would like to back up the volume to local ...
user avatar
  • 33
0 votes
1 answer
48 views

Ceph connect to local node

I have an idea for an application that I'd like to build and one of the requirements is a globally replicated filesystem. Things like Ceph and GlusterFS exist, but I'm not sure they meet my particular ...
user avatar
0 votes
1 answer
397 views

cannot Delete objects on s3 ceph bucket

I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
user avatar
0 votes
0 answers
50 views

Reduced ceph storage pool size

Currently running a 7-node Ceph cluster, used as file system storage. Two independent nodes are used to run the mon+mgr+mds service, the other 5 nodes are used as storage nodes and one of the storage ...
user avatar
3 votes
0 answers
189 views

How to create Ceph OSDs on Partitions using cephadm, on ceph-pacific?

In the pacific version of Ceph, OSD creation is not allowing by cephadm on partitions.(We have NvMes with partitions). The command used is: ceph orch daemon add osd <host>:<device-path> ...
user avatar
0 votes
0 answers
105 views

nfs (or ceph, or k8s) works strange: ls/tree not working, but i can read files

I have some problems and i want to know where i have to check. my env: ceph nfs mount to a k8s pod problem: at mounted path (on k8s node and inside the pod too), ls/tree not working, but i can read/...
user avatar
  • 1
0 votes
1 answer
306 views

Multiple port on the same server behind the same backend

I’m using haproxy with my ceph cluster and I’ve created more gateways on 1 server with different ports. I’ve added to the backend configuration with the different port numbers but seems like haproxy ...
user avatar
  • 15
0 votes
1 answer
442 views

How can I extend or resize lvm from disk space

We use ceph to manage the storage and recently increased /dev/sdb's disk size. I'm not sure how to utilize the newly available space in my ceph's lvm. sdb ...
user avatar
  • 1
2 votes
0 answers
74 views

CEPH/ RGW encryption with mutliple tenants

I'm currently planning a small OpenStack deployment and I want to use Ceph for object storage (via Rados Gateway) and block storage. Ceph supports multitenancy. It also supports encryption using key ...
user avatar
1 vote
1 answer
736 views

bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted

When I have upgrade the from Octopus 15.2.10 to Pacific 16.2.0, the mon nodes starts successfully using the manual upgrade process (by installing the packages with no orch) however when I have upgrade ...
user avatar
  • 27
2 votes
1 answer
719 views

Orchestrator is not available with fresh Rook instance

I'm trying to set up a rook Ceph cluster on my kubernetes cluster. Topography: 3 kubernetes nodes (all are master/worker pods) Each node has /dev/vdX on it for ceph Each node is intended to work as ...
user avatar
  • 613
0 votes
1 answer
305 views

Multi site ceph + proxmox

We have 2 proxmox clusters in production. They contain Ceph cluster with cephfs is on it. The task is to sync files between sites. Site A is master and site B is for backup. On site A cephfs files are ...
user avatar
0 votes
1 answer
353 views

How to know how many files can Ceph delete per 1 minute?

As the question says. Is there a way to find out how many files can Ceph delete per 1 minute? A little backstory: Image hosting website, running Ceph FS on 8TB WD RED 3.5 HDD's, around 51 TiB already ...
user avatar
0 votes
0 answers
801 views

Cinder volume service goes down while creating/deleting multiple volumes at the same time

Environment: Openstack Rocky Ceph Mimic openstack-cinder-13.0.8-0 Cinder volume status goes down while creating or deleting multiple volumes at the same time manually or while Rally test is running. ...
user avatar
0 votes
1 answer
1k views

Ceph Cluster - Reduced data availability: 96 pgs inactive And All OSD Nodes Are Down

I set up my Ceph Cluster by following this document. I have one Manager Node, one Monitor Node, and three OSD Nodes. The problem is that right after I finished setting up the cluster, the ceph health ...
user avatar
0 votes
1 answer
426 views

Pipe rbd export comand to tar to create compressed export

Is it possible to pipe the output of rbd export directly to something else like tar to create a compressed archive of the .img file without having to write to disk first? Something like rbd export ...
user avatar
  • 613
1 vote
1 answer
796 views

Ceph: is setting lower "size" parameter on a live pool possible?

I have a 5-node Proxmox cluster using Ceph as the primary VM storage backend. The Ceph pool is currently configured with a size of 5 (1 data replica per OSD per node) and a min_size of 1. Due to the ...
user avatar
0 votes
1 answer
134 views

Openstack Using Ceph Erasure Code Pool

I tried configuring openstack with ceph pool erasure code backend I'm trying to using —data-pool on ceph.conf using rbd default data pool like the following blog https://themeanti.me/technology/2018/...
user avatar
1 vote
1 answer
236 views

Recommended values for mon osd down out in ceph

Are there any recommended values for "mon osd down out"? The cluster has 15 nodes and 300 osds. It's running ceph Jewl. The default value in Jewel is 300 seconds. I was wondering, are there ...
user avatar
0 votes
2 answers
1k views

Proxmox Ceph - Got timeout on separate network

I've installed on 4 nodes a completly fresh OS with Proxmox. Every node has 2xNVMe und 1xHD, one NIC public, one NIC private. On the public network there is an additional wireguard interface running ...
user avatar
  • 408
1 vote
0 answers
245 views

how to sync ceph bucket with scaleway s3 bucket or any s3 bucket for?

i,m trying to sync a ceph cluster to a scaleway s3 bucket i have the secret, access token and https endpoint for my s3 bucket ceph version: mimic i followed the cloud sync module guide at the ...
user avatar
  • 83
1 vote
0 answers
61 views

Ceph-ansible: using the OS-SSD as cache

In ceph-ansible, you can specify a disk to be used for databases in the hosts file using the dedicated_devices directory. Example: hosts: stor01.hostname.com devices: ['/dev/sdd', '/dev/...
user avatar
  • 111
3 votes
1 answer
3k views

Ceph e5 handle_auth_request failed to assign global_id after a host outage

I have a small 3-host Ceph cluster with Ubuntu 20.04.1 and Ceph 15.2.5 using docker containers and deployed with cephadm. Yesterday one of the hosts (s65-ceph) had a power outage. The other two ...
user avatar
0 votes
1 answer
118 views

ceph-deploy install --release fails to upgrade on debian stretch

this is my first post on serverfault! I'm trying to setup a test ceph cluster to see if it can meet our storage needs. I've been following the instructions here I ran into some permissions issues when ...
user avatar
0 votes
3 answers
737 views

Data resiliency in Ceph or Gluster

Say I am most concerned with keeping my data safe from physical hard drive failure and the computer it connects to directly. RAID1 and RAID5 I understand. They provide protections against data loss ...
user avatar
0 votes
1 answer
203 views

Memory level for Ceph logs. What's this for?

I want to increase the logging level of some OSDs, but I don't understand what the memory level is for. I didn't find an exact explanation in the documentation. Is there a need to increase this level? ...
user avatar
1 vote
0 answers
1k views

CEPH cluster became unresponsive: e5 handle_auth_request failed to assign global_id

I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 hosts in total, all ESXi VMs). It lived through a couple of reboots without problem, then I've reconfigured the main host a bit: ...
user avatar
  • 65
1 vote
1 answer
88 views

Can my Ceph nodes have the same hostname?

I have a high availability solution which requires me to have all my nodes using the same hostname. I've been tasked to use ceph cluster storage, but I just reached a dead end. I can't figure out how ...
user avatar
3 votes
2 answers
704 views

How to best use large NVMe array for VMWare Datastore

I recently acquired a Supermicro SYS-1029P-N32R with 16 14TB NVMe SSD's (Micron 9300 Pro) at work. We have 100 gigabit networking on the box to our VMWare hosts. We've tried using FreeNAS to host ZFS ...
user avatar
3 votes
0 answers
86 views

Ceph Rados GW Proxy

I've been given an access key to a Ceph cluster that runs radoswgw to provide S3. The key allows bucket creation, object reading etc. I don't admin the Ceph or radosgw. I have a group of users that ...
user avatar
  • 131
0 votes
1 answer
595 views

Does Kubeadm reset delete persistent volumes

I need to change the advertise address of my Kubernetes master node. I can do this with a kubeadm reset, but I don't want to lose my (rook-ceph) persistent volumes on the master node. Is it safe to do ...
user avatar
0 votes
1 answer
59 views

OpenStack redundant shared storage backend solution

Which storage backends are recommended for OpenStack which allows live migration and shared storage? Is it NFS / Ceph / storage backend drivers to commercial storage solutions? Which of these is ...
user avatar
  • 98
0 votes
1 answer
380 views

Error running ceph-deploy install

I have been following the following tutorial to setup ceph on CentOS https://wiki.centos.org/SpecialInterestGroup/Storage/ceph-Quickstart When I try to run ceph-deploy to install to my other nodes, ...
user avatar
  • 2,633
-2 votes
1 answer
71 views

Backup solution for medium infrastructure with commodity hardware

I'm looking for recommendations for a backup solution for our Infrastructure. We have lots of old Hardware, which is not very fast but just laying around. We have like 20 Servers for backup with 8-...
user avatar
1 vote
1 answer
2k views

Poor performance with rook, ceph and RBD

I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD. The OSDs are using the same disk as the VM Operating System. The VM disks are remote (...
user avatar
0 votes
1 answer
6k views

Ceph hardware sizing calculator

I would like to calculate the Hardware sizing for a ceph cluster. There are so few references to the sizing, that I try to get these details here in the community. E.g. what shall i have depending on ...
user avatar
  • 277