Questions tagged [ceph]

Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.

Filter by
Sorted by
Tagged with
0 votes
0 answers
50 views

Setting Read-Only Access On Ceph Rgw Bucket For Another User On Same Tenant Not Working

Two users can't see to see each other's buckets on Ceph Reef. First, I create the users on a ceph admin vm: radosgw-admin user create --tenant=homelab --uid=user1 --display-name="First User" ...
Magnitus's user avatar
  • 101
2 votes
1 answer
177 views

Where does CEPH store device metrics for each device? [ceph reef]

I just deployed my first ceph cluster and have been trying multiple things so far. Though, I couldn't understand how ceph store metric for the devices. I can easily access those through: # ceph device ...
Robson Jr's user avatar
0 votes
1 answer
63 views

Completely Reprovisioning Ceph Rgw Without Reprovisioning Ceph Cluster

I recently began learning about Ceph and setup my own ansible scripts to setup a Ceph cluster (yes, I'm aware that cephadm-ansible exists, but I want to get comfortable managing ceph). Initially, I ...
Magnitus's user avatar
  • 101
0 votes
0 answers
37 views

why rook cephfs storage class store pv data in 2 places?

I am a newbie to ceph and run ceph using rook in bare metal environment.my question is : when you use rook cephfs provisioner as storage class and create a pvc to assign to a pod , when the pv is ...
Ali farahzadi's user avatar
1 vote
0 answers
130 views

Deleting file in ceph does not free up disk space

Ceph version: 16.2.13 (pacific) (I know that pacific is deprecated but the whole environment is legacy (like centos 7.3). I am not authorized to upgrade as well.) The cluster has 6 servers (22 osds, ...
SimpleProgrammer's user avatar
0 votes
0 answers
23 views

Ceph rbd commands are hanging forever when executed on remote server

I have an openstack compute node connected to ceph cluster with 10G. The compute can ping and reach the all ceph nodes and port 3300 but somehow rbd commands are hanging which is stopping the compute ...
student0909's user avatar
0 votes
0 answers
31 views

openstack fails to create an instance after integrating with rbd ceph

I have configured openstack kolla to enable ceph for glance, cinder and nova. I must say that cinder and glance are working perfectly fine with ceph. Nova on the other hand refuses to work properly. ...
student0909's user avatar
0 votes
0 answers
32 views

openstack nova goes down when ceph pool is set to have more than 1 pgs

I have openstack connected to ceph cluster. When i increase the pgs of the vms pool to be more than 1 and launch instance in openstack the instance is stuck on spawning state and nova compute ...
Philip Scot's user avatar
-1 votes
1 answer
160 views

Can I create buckets in ceph object storage using a command-line interface?

Is it possible to create buckets in the ceph object store via the rados gateway from a CLI? I am aware of the radosgw-admin tool, however, it doesn't appear to have this ability. It appears mainly to ...
rafiki's user avatar
  • 9
0 votes
0 answers
15 views

cephfs mount: can't open files with LibreOffice

I'm able to open/write different relevant filetypes with other applications (vim,code,gedit,...) from remote cephfs storage. libreoffice has been configured to prevent file-locks. All dirs and files ...
Peedy2495's user avatar
0 votes
1 answer
39 views

ceph snapshots are not running

Our ceph cluster (quincy 17.2.7) runs in a docker container. The snapshot schedule was created as: ceph fs snap-schedule add / 1h 2023-12-13T09:00:00 nerfcomputing2024 and retention: ceph fs snap-...
gmaggi's user avatar
  • 1
0 votes
0 answers
163 views

RADOS permission denied

I'm trying to install a ceph cluster using 3 ubuntu 20.04 machine each with 4 disk. I've followed the manual installation from here https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-...
fraschizzato's user avatar
0 votes
1 answer
56 views

how number of replicas is aligned with crush rule in a ceph cluster

I have some difficulties to understand whether the crush rule is aligned with the number of replicas. So, imagine I create a crush rule named replicate_in_2_racks as follows: ceph osd crush rule ...
gmaggi's user avatar
  • 1
0 votes
2 answers
173 views

how to get the used space (size in bytes) of a directory in a ceph file system

I have a ceph cluster quincy 17.2.7 I wonder if ceph has some tools to quickly get the the used space (size in bytes) of a directory in a ceph file system. I know one can get that using du -hs /fs/dir-...
gmaggi's user avatar
  • 1
0 votes
0 answers
90 views

"ceph-volume lvm new-db" does not seem to perform as should be

A database device seems to be added to the OSD (via ceph-volume lvm new-db), despite an error is thrown. Until now, it is unclear to me if the DB device was actually successfully added to the OSD or ...
gmaggi's user avatar
  • 1
0 votes
0 answers
238 views

Unable to mount CephFS on client - Rocky8.8

When attempting to mount a Ceph file system the mount command times out: mount error 110 = Connection timed out dmesg shows the following: ceph: device name is missing path (no : separator in splunk@$...
CJW's user avatar
  • 101
0 votes
1 answer
46 views

ceph remove spurious data

I’d like to remove some “spurious" data: root@XX03:/# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0 TOTAL ...
gmaggi's user avatar
  • 1
0 votes
0 answers
121 views

Ceph mounted rbd poor performance

all tests are done from an external server connected with 10G to all 3 ceph nodes and the cluster is all nvme flash. when i ran fio with this config: [global] ioengine=rbd clientname=admin pool=...
Christian Willing's user avatar
0 votes
1 answer
252 views

Now that the in-tree Kubernetes CephFS persistent volume plugin is deprecated how can I use Rook to get the same mounting behavior?

As of Kubernetes v1.28 the CephFS in-tree persistent volume plugin is deprecated. The official suggestion is to now instead use the third-party CSI driver. However, I am not sure how I can use for ...
rafiki's user avatar
  • 9
0 votes
0 answers
131 views

I'm trying to expose a folder size in Node exporter but messes up. any suggestions?

This wasn't the case for a few months but suddenly my Node exporter in the server service messed up. Oct 02 10:13:01 nodefs01 node_exporter[117252]: time="2023-10-02T10:13:01Z" level=error ...
user3324696's user avatar
1 vote
1 answer
222 views

Kubernetes Rook-Ceph: different pools for different use-cases?

I'm experimenting with rook-ceph on kubernetes for storage again. I have 3 nodes, which each have a local ssd-partition mounted at /dev/sda2, and a NAS-volume at /dev/sdb. In my rook-config, I use ...
Lars's user avatar
  • 458
0 votes
2 answers
74 views

how to use the images that have been create by "Offload rbd’s copy_volume_to_image" method in openstack?

I have an openstack integrated with ceph. I'm working on the fastest way to create a copy of an instance's volume without using cinder_backup module. in the Ceph it is possible to access the rbd files ...
sina's user avatar
  • 3
0 votes
1 answer
109 views

extract openstack instances' volume from ceph

I have an OpenStack-kolla environment with the Ceph backend. I've configured my cinder backend with NFS. I can create a Backup with Cinder from all instances and store them on NFS. In this case, I don'...
sina's user avatar
  • 3
1 vote
1 answer
144 views

Ceph: Rados Gateway web endpoint not responding

I'm creating a simple Ceph cluster and trying to connect to Ceph Gateway. This is ceph status output of my Ceph cluster: cluster: id: a7f64266-0894-4f1e-a635-d0aeaca0e993 health: ...
izarc's user avatar
  • 11
0 votes
0 answers
55 views

Upgrade OS Bionic to Focal which includes Openstack and Ceph cluster

So i have openstack and ceph cluster with 3 controllers and 3 computes. This is the information about my cluster : OS : ubuntu bionic Openstack kolla-ansible : Ussuri 10.4.0 Ceph ceph-ansible : ...
Adelia Nurlina's user avatar
0 votes
1 answer
444 views

kolla-ansible openstack bootstrap-servers failed on enabling ceph apt repository (ubuntu 22.04)

This is to deploy the recent kolla-ansible openstack. The problem happens since last week. I retested it on a good environment successfully deployed on 7/14/2023 and got bootstrap-servers failed ...
user1039217's user avatar
0 votes
0 answers
210 views

Ceph's failure to automount after network failure

I'm having some problems with the mounting of a ceph-cluster on debian machines, don't know if I'm doing something wrong, if it's a version problem or anything else. I'm using the ceph cluster from ...
Tio's user avatar
  • 165
0 votes
0 answers
114 views

ceph - is a drive mapping change recoverable /dev/sdi /dev/sdj

I was testing drive hot plugging on my server, and unplugged one of the drives, then plugged it back in. Though, when plugged back in the drive letter changed from /dev/sdj to /dev/sdl. Now ceph will ...
donkeyx's user avatar
  • 101
0 votes
0 answers
48 views

Ceph File system OS installation

I'm building a Ceph cluster. I've been reading for months and I can't find this information. I am going to use Ubuntu 20.04 What is the file system that I have to use for the installation of the ...
Fernando Grubicy's user avatar
0 votes
1 answer
1k views

ceph "permission denied" when adding storage OSD

I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the ...
user5717448's user avatar
3 votes
1 answer
126 views

Pooled storage with varying redundancy per file system

I have some files that I want stored mirrored. I have some files that I only need single copies (i.e. scratch data, easily regenerated data, etc). And I have some files that are so critical, I want it ...
David Pfeffer's user avatar
0 votes
1 answer
71 views

OSD stuck in booting with messenger component log: Operation not permitted error

After rebooting k8s nodes, OSD didn't join the cluster with errors related to authentication. I have added them to auth list and that error disappears. Now OSD nodes join the cluster but they don't ...
Ahmad Ahmadi's user avatar
0 votes
1 answer
310 views

Ceph - reset ceph osd_max_backfills to default value

I am attempting to reset some backfill settings to defaults after having changed them to slow down the backfilling. I am struggling to work out how to do it though: This example shows that the ...
donkeyx's user avatar
  • 101
0 votes
1 answer
457 views

Ceph determine PG based on rados object name

In Ceph, the command rados --pgid <pgid> ls allows you to list all the rados objects that are stored in a specific placement group. My question is: is there a command that does the reverse? So, ...
roelvanmeer's user avatar
  • 1,784
1 vote
0 answers
143 views

Error deploying Ceph Quincy using ceph-ansible 7 on Rocky 9

I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am getting some problems and I don't know where to search for the reason. PS : I did the same deployment on Rocky8 using ceph-ansible ...
Wodel's user avatar
  • 51
0 votes
0 answers
414 views

changing queue parameter nr_request on NVMe dev

We would like to use NVMes (SAMSUNG MZ1LB1T9HALS-00007) as a cache-tier in a Ceph System (17.2.0). The problem is that the stored data has a lot of small files and if hot-tiering is set to ~70% on the ...
pawel_winzig's user avatar
0 votes
2 answers
2k views

Which Ceph OSD is mapped to which pysical drive

I have a Ceph system with 8 OSD's and 8 disks mapped 1:1. One of the disks is giving me smart errors and I would like to replace it. How do I know which physical disk is mapped to which OSD?
Mr. Diba's user avatar
  • 128
0 votes
1 answer
193 views

ceph-deploy install command fails with [ceph_deploy][ERROR ] RuntimeError: configparser.NoSectionError: No section: 'main'

Command ceph-deploy install admin datanode_dn2 fails with output: [ceph_deploy.install][INFO ] Distro info: rocky 9.1 blue onyx [admin][INFO ] installing Ceph on admin [admin][INFO ] Running ...
Kabira  K's user avatar
  • 133
3 votes
2 answers
2k views

Ceph does not detect disks

I am using cephadm. I had a test cluster up and running. I re-installed the OS and during installation nuked the drives by deleting the LVM volume groups and partitions. Now lsblk shows the devices to ...
Tintin's user avatar
  • 173
1 vote
0 answers
2k views

Ceph reading and writing performance problems, fast reading and slow writing

Ceph reading and writing performance problems, fast reading and slow writing Hello, we need to migrate all cloud environments to Proxmox. At present, I am evaluating and testing Proxmox+Ceph+OpenStack....
Curtis James's user avatar
1 vote
2 answers
231 views

Configure ceph block storage pool in cockpit

I have a plain install on machine C with Oracle Linux 9.1 and cockpit and cockpit-machines. On machines A and B I have a ceph cluster configured which defines an rbd block storage pool for VM disks. ...
a.ilchinger's user avatar
0 votes
1 answer
203 views

Cache mechanisme in kubernetes

I've got the next setup: Proxmox 7.2 CEPH 16.2.9 K3S v1.23.15+k3s1 CEPH CSI v3.7.2 CEPH using as RBD-storage for QEMU images and K8S PVC. When I do disk benchmark in QEMU I've got the next results: ...
chs_voks's user avatar
-1 votes
1 answer
546 views

Cephadm + Openstack Keystone Integration

Hello I am trying to Integrate Openstack Keystone as Ceph authentication Mechanism So i can use ceph object storage as openstack swift backend Environment: Kernel : Ubunutu Server LTS 22.04 (minimal) ...
Hoodad Tabibi's user avatar
0 votes
1 answer
1k views

Why is ceph is not detecting ssd device on a new node?

I have installed a ceph cluster (quincy) with already 2 nodes and 4 OSDs. Now I added a 3rd host running Debian (bullseye) to the cluster. The new host is deteced correctly and runs a mom. The problem ...
Ralph's user avatar
  • 101
0 votes
1 answer
109 views

Enabling Cephx In cephadm depolyment

Hello i am trying to enable cephx in my ceph cluster i added this three lines to /etc/ceph/ceph.conf auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx i didn'...
Hoodad Tabibi's user avatar
1 vote
1 answer
823 views

Ceph alertmanager config

I've installed ceph using cephadm Including monitoring stack prometheus, alertmanager, node-exporter Currently I'm trying to add a telegram receiver (Telegram is supported from v0.24.0, so I've ...
SergkeiM's user avatar
1 vote
0 answers
62 views

Ceph reports bucket space utilization and total cluster utilization that is inconsistent

I copied the contents of an older Ceph cluster to a new Ceph cluster using rclone. Because several of the buckets had tens of millions of objects in a single directory I had to enumerate these ...
wilsotc's user avatar
  • 111
0 votes
1 answer
105 views

In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules? [duplicate]

I'm designing a ceph cluster for mixing cephFS and rbd for my company( VM et file infrastructure). In my set I need 32To raw storage by node. I start with 5 nodes. The seller quotes propose me to ...
Ztevoz Milloz's user avatar
1 vote
0 answers
886 views

How to configure a CEPH host with public and private NICs

So here is the situation. I am deploying CEPH on a public cloud using cephadm. I am trying to create a private,public network config as described here First we bootstrap the monitor with ceph ...
mohamed emad's user avatar
0 votes
1 answer
200 views

Is running CEPH on public cloud without data replication a good idea?

My startup is thinking about adopting CEPH for block storage (and other features it provides). Most of our infrastructure runs on public cloud (DO,GCP,AWS,etc...). On public cloud data storage you ...
mohamed emad's user avatar