As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [ceph]

Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

ceph "permission denied" when adding storage OSD

I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the ...
user5717448's user avatar
2 votes
1 answer
25 views

Pooled storage with varying redundancy per file system

I have some files that I want stored mirrored. I have some files that I only need single copies (i.e. scratch data, easily regenerated data, etc). And I have some files that are so critical, I want it ...
David Pfeffer's user avatar
0 votes
1 answer
19 views

OSD stuck in booting with messenger component log: Operation not permitted error

After rebooting k8s nodes, OSD didn't join the cluster with errors related to authentication. I have added them to auth list and that error disappears. Now OSD nodes join the cluster but they don't ...
Ahmad Ahmadi's user avatar
0 votes
0 answers
73 views

Ceph - reset ceph osd_max_backfills to default value

I am attempting to reset some backfill settings to defaults after having changed them to slow down the backfilling. I am struggling to work out how to do it though: This example shows that the ...
donkeyx's user avatar
  • 101
0 votes
1 answer
82 views

Ceph determine PG based on rados object name

In Ceph, the command rados --pgid <pgid> ls allows you to list all the rados objects that are stored in a specific placement group. My question is: is there a command that does the reverse? So, ...
roelvanmeer's user avatar
  • 1,754
1 vote
0 answers
60 views

Error deploying Ceph Quincy using ceph-ansible 7 on Rocky 9

I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am getting some problems and I don't know where to search for the reason. PS : I did the same deployment on Rocky8 using ceph-ansible ...
Wodel's user avatar
  • 51
0 votes
0 answers
168 views

changing queue parameter nr_request on NVMe dev

We would like to use NVMes (SAMSUNG MZ1LB1T9HALS-00007) as a cache-tier in a Ceph System (17.2.0). The problem is that the stored data has a lot of small files and if hot-tiering is set to ~70% on the ...
pawel_winzig's user avatar
0 votes
2 answers
215 views

Which Ceph OSD is mapped to which pysical drive

I have a Ceph system with 8 OSD's and 8 disks mapped 1:1. One of the disks is giving me smart errors and I would like to replace it. How do I know which physical disk is mapped to which OSD?
Mr. Diba's user avatar
  • 128
0 votes
1 answer
70 views

ceph-deploy install command fails with [ceph_deploy][ERROR ] RuntimeError: configparser.NoSectionError: No section: 'main'

Command ceph-deploy install admin datanode_dn2 fails with output: [ceph_deploy.install][INFO ] Distro info: rocky 9.1 blue onyx [admin][INFO ] installing Ceph on admin [admin][INFO ] Running ...
Kabira  K's user avatar
  • 133
0 votes
1 answer
375 views

Ceph does not detect disks

I am using cephadm. I had a test cluster up and running. I re-installed the OS and during installation nuked the drives by deleting the LVM volume groups and partitions. Now lsblk shows the devices to ...
Tintin's user avatar
  • 101
0 votes
0 answers
535 views

Ceph reading and writing performance problems, fast reading and slow writing

Ceph reading and writing performance problems, fast reading and slow writing Hello, we need to migrate all cloud environments to Proxmox. At present, I am evaluating and testing Proxmox+Ceph+OpenStack....
Curtis James's user avatar
1 vote
2 answers
78 views

Configure ceph block storage pool in cockpit

I have a plain install on machine C with Oracle Linux 9.1 and cockpit and cockpit-machines. On machines A and B I have a ceph cluster configured which defines an rbd block storage pool for VM disks. ...
a.ilchinger's user avatar
0 votes
1 answer
89 views

Cache mechanisme in kubernetes

I've got the next setup: Proxmox 7.2 CEPH 16.2.9 K3S v1.23.15+k3s1 CEPH CSI v3.7.2 CEPH using as RBD-storage for QEMU images and K8S PVC. When I do disk benchmark in QEMU I've got the next results: ...
chs_voks's user avatar
-1 votes
1 answer
281 views

Cephadm + Openstack Keystone Integration

Hello I am trying to Integrate Openstack Keystone as Ceph authentication Mechanism So i can use ceph object storage as openstack swift backend Environment: Kernel : Ubunutu Server LTS 22.04 (minimal) ...
Hoodad Tabibi's user avatar
0 votes
1 answer
396 views

Why is ceph is not detecting ssd device on a new node?

I have installed a ceph cluster (quincy) with already 2 nodes and 4 OSDs. Now I added a 3rd host running Debian (bullseye) to the cluster. The new host is deteced correctly and runs a mom. The problem ...
Ralph's user avatar
  • 101
0 votes
1 answer
60 views

Enabling Cephx In cephadm depolyment

Hello i am trying to enable cephx in my ceph cluster i added this three lines to /etc/ceph/ceph.conf auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx i didn'...
Hoodad Tabibi's user avatar
0 votes
0 answers
298 views

Unable to start Ceph RGW service

Context: I'm trying to setup a new ceph-17.2.5 cluster Initial setup consists of 1 mgr, 1 mon and 1 osd all installed on same box using cephadm. So far I am able to access http://192.168.3.50:8080/api/...
Karan Shukla's user avatar
1 vote
1 answer
306 views

Ceph alertmanager config

I've installed ceph using cephadm Including monitoring stack prometheus, alertmanager, node-exporter Currently I'm trying to add a telegram receiver (Telegram is supported from v0.24.0, so I've ...
SergkeiM's user avatar
1 vote
0 answers
25 views

Ceph reports bucket space utilization and total cluster utilization that is inconsistent

I copied the contents of an older Ceph cluster to a new Ceph cluster using rclone. Because several of the buckets had tens of millions of objects in a single directory I had to enumerate these ...
wilsotc's user avatar
  • 111
0 votes
1 answer
63 views

In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules? [duplicate]

I'm designing a ceph cluster for mixing cephFS and rbd for my company( VM et file infrastructure). In my set I need 32To raw storage by node. I start with 5 nodes. The seller quotes propose me to ...
Ztevoz Milloz's user avatar
0 votes
0 answers
64 views

Dedicated Network connection for CEPH or aggregated general purpose network connection?

We are building a Kubernetes Cluster on bare metal. As the cluster is getting physically installed, I was asked to weigh in on the following question: Each node has two network connections of 25Gbps. ...
bountrisv's user avatar
0 votes
0 answers
75 views

Where do I find a script to check a Ceph iSCSI gateway configuration?

On a Ceph cluster with iSCSI gateway we have an issue with gateway configuration changes being applied partially. The result is a broken gateway.conf, which goes unnoticed until (possibly weeks later) ...
reinierpost's user avatar
1 vote
0 answers
516 views

How to configure a CEPH host with public and private NICs

So here is the situation. I am deploying CEPH on a public cloud using cephadm. I am trying to create a private,public network config as described here First we bootstrap the monitor with ceph ...
mohamed emad's user avatar
0 votes
1 answer
123 views

Is running CEPH on public cloud without data replication a good idea?

My startup is thinking about adopting CEPH for block storage (and other features it provides). Most of our infrastructure runs on public cloud (DO,GCP,AWS,etc...). On public cloud data storage you ...
mohamed emad's user avatar
0 votes
0 answers
50 views

Serve cephFS files over HTTP (or S3?)

I have a CEPH+LXD cluster configuration. And I would like to serve the files in the container over HTTP (S3,whatever doesn't matter). Is there a way to do this using the rgw service? Like can I point ...
mohamed emad's user avatar
0 votes
0 answers
118 views

Volume not getting mounted (Nexus is running on this Pod)

Nexus is running on worker node and Nexus UI went down because of storage issue. We increased the storage space from 150 GB to 210GB for PV & PVC. Increased storage for PV and PVC Then, when we ...
Shree's user avatar
  • 1
0 votes
0 answers
230 views

Ceph : Deleted bucket still exist

I have tested multiple configuration of pool and created multiple placement target. However i had deleted placement group and pools (index, non-ec and data pool) before deleting the bucket. Since this ...
Thomas Coche's user avatar
0 votes
1 answer
640 views

Remove orphaned Ceph volumes

I have installed a Ceph cluster across three nodes which needed completely reinstalling after testing. It seems that on one of the nodes, some configuration data remained which Ceph is still picking ...
Chris's user avatar
  • 1,289
0 votes
0 answers
234 views

Usage stats of CephFS subvolumes

Is there a way to gather and show the disk usage stats of CephFS subvolumes? Is there anything similar to rbd du for CephFS subvolumes? The only idea I have is something really ugly: filesystem="...
Volodymyr Melnyk's user avatar
1 vote
0 answers
111 views

Computing stripe count for erasure coded storage

I'm setting up a ceph cluster (first time for me) which in the end will be made of ~100 disks spread over 10 hosts. I'm going with a single erasure coded data pool to maximize disk space; my ...
Nicola Mori's user avatar
0 votes
1 answer
249 views

Ceph RGW 16.2.7 CLI changes

I am trying to run a new Ceph cluster with Rados GW using the last software version 16.2.7, but when I set up RGW nodes I found out there are some changes in the CLI comparing with a version 16.2.4 I ...
Aleksandr Makhov's user avatar
0 votes
0 answers
324 views

Openstack Glance Image Corrupted

I have an openstack instance up and running for 4 months and would like to migrate it to another nova node. When I click migrate instance option of that 4 month running instance on openstack dashboard....
ony4869's user avatar
  • 33
0 votes
2 answers
9k views

ssh-copy-id ceph@osd-0 /bin/ssh-copy-id: ERROR: failed to open ID file '/home/ceph/.pub': No such file or directory

I can ssh successfully my ceph osd nodes but when i am trying to copy ssh id i am getting below error can anyone guide why i am getting below errors. [ceph@monitor ~]$ ssh-copy-id ceph@osd-0 /bin/ssh-...
Noor's user avatar
  • 11
2 votes
0 answers
2k views

Proxmox on Ceph performance & stability issues / Configuration doubts

We have just installed a cluster of 6 Proxmox servers, using 3 nodes as Ceph storage, and 3 nodes as compute nodes. We are experiencing strange and critical issues with the performances and stability ...
Danyright's user avatar
  • 163
1 vote
0 answers
301 views

Ceph RGW: slow `list_bucket` requests

I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds: $ ...
Taha Jahangir's user avatar
0 votes
1 answer
281 views

Changing Ceph to store number of data copies

Currently I am using replication for data placement and think I am using three copies, which I think is the default. How do I change ceph config to store 4 copies on different nodes in different ...
Kampton Joe's user avatar
0 votes
1 answer
308 views

Three Single drive Raid 1 vs JBOD

I have an intel server of which the RAID Controller does not support JBOD mode (same problem as my Dell R820s). But I want to do Ceph, so I "need" a JBOD. So I set my 3 SSDs as single drive ...
Johnride's user avatar
  • 101
3 votes
0 answers
378 views

Openstack Nova and Ceph Volume Attachment Issue

I am trying out the volume attachment function at Openstack (version:wallaby) to the server as additional device but failed. The volume backend is ceph which all of the osds are up and healthy. ceph-...
ony4869's user avatar
  • 33
0 votes
1 answer
72 views

Openstack Wallaby Volume Backup Issue

Recently I am testing kubernetes with 1 master node and 3 worker nodes that built on Openstack(version: Wallaby). It involves lots of testing and therefore I would like to back up the volume to local ...
ony4869's user avatar
  • 33
0 votes
1 answer
146 views

Ceph connect to local node

I have an idea for an application that I'd like to build and one of the requirements is a globally replicated filesystem. Things like Ceph and GlusterFS exist, but I'm not sure they meet my particular ...
Jack Slingerland's user avatar
0 votes
1 answer
682 views

cannot Delete objects on s3 ceph bucket

I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
AliReza NaSRi's user avatar
3 votes
0 answers
449 views

How to create Ceph OSDs on Partitions using cephadm, on ceph-pacific?

In the pacific version of Ceph, OSD creation is not allowing by cephadm on partitions.(We have NvMes with partitions). The command used is: ceph orch daemon add osd <host>:<device-path> ...
Arun Vinod's user avatar
0 votes
0 answers
192 views

nfs (or ceph, or k8s) works strange: ls/tree not working, but i can read files

I have some problems and i want to know where i have to check. my env: ceph nfs mount to a k8s pod problem: at mounted path (on k8s node and inside the pod too), ls/tree not working, but i can read/...
whojes's user avatar
  • 1
0 votes
1 answer
881 views

Multiple port on the same server behind the same backend

I’m using haproxy with my ceph cluster and I’ve created more gateways on 1 server with different ports. I’ve added to the backend configuration with the different port numbers but seems like haproxy ...
Badb0y's user avatar
  • 115
0 votes
1 answer
803 views

How can I extend or resize lvm from disk space

We use ceph to manage the storage and recently increased /dev/sdb's disk size. I'm not sure how to utilize the newly available space in my ceph's lvm. sdb ...
rafas's user avatar
  • 1
2 votes
0 answers
100 views

CEPH/ RGW encryption with mutliple tenants

I'm currently planning a small OpenStack deployment and I want to use Ceph for object storage (via Rados Gateway) and block storage. Ceph supports multitenancy. It also supports encryption using key ...
Frederic Schönberger's user avatar
2 votes
1 answer
1k views

bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted

When I have upgrade the from Octopus 15.2.10 to Pacific 16.2.0, the mon nodes starts successfully using the manual upgrade process (by installing the packages with no orch) however when I have upgrade ...
user393397's user avatar
2 votes
1 answer
2k views

Orchestrator is not available with fresh Rook instance

I'm trying to set up a rook Ceph cluster on my kubernetes cluster. Topography: 3 kubernetes nodes (all are master/worker pods) Each node has /dev/vdX on it for ceph Each node is intended to work as ...
cclloyd's user avatar
  • 593
0 votes
1 answer
434 views

Multi site ceph + proxmox

We have 2 proxmox clusters in production. They contain Ceph cluster with cephfs is on it. The task is to sync files between sites. Site A is master and site B is for backup. On site A cephfs files are ...
Andrejs Snorins's user avatar
0 votes
1 answer
732 views

How to know how many files can Ceph delete per 1 minute?

As the question says. Is there a way to find out how many files can Ceph delete per 1 minute? A little backstory: Image hosting website, running Ceph FS on 8TB WD RED 3.5 HDD's, around 51 TiB already ...
En0w1na's user avatar