Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
202
questions
3
votes
0
answers
55
views
CEPH Quincy installation with multipathd enabled
we have a setup comprising of 6 servers hosting CentOS 8 Minimal Installation with CEPH Quincy version 18.2.2 supported by 20Gbps fiber optics NICs and a dual Xeon Intel processors, bootstrapped the ...
- 31
2
votes
1
answer
70
views
Ceph cluster mixing nvme disks with different sizes
I am deploying a ceph cluster, the cluster will have three controller servers and 27 osd nodes.
Each osd node has 3x3.8Tb nvme + 1x1.9Tb nvme disks, for a total of 4 nvme disks per node.
The failure ...
- 2,020
0
votes
0
answers
24
views
Ceph s3 LDAP not working despite following instructions
I have a problem with LDAP configuration in Ceph. I have a fresh cluster (for testing purposes). No operation has been performed on it except adding certificates to Dashborad and Grafana and issuing . ...
- 11
0
votes
1
answer
75
views
How to enable dm-crypt on Ceph OSD with cephadm?
I had originally used the following website (https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-servers/) to stand up a Ceph cluster on Ubuntu 22.04 with Cephadm and Ansible. ...
- 467
0
votes
0
answers
110
views
What is the proper way to setup Rados Gateway (RGW) under Ceph using Cephadm?
I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a lab environment. I have buckets and pools already setup. The ceph.io site is confusing on how to setup the Rados Gateway (radosgw) ...
0
votes
0
answers
118
views
Setting Read-Only Access On Ceph Rgw Bucket For Another User On Same Tenant Not Working
Two users can't see to see each other's buckets on Ceph Reef.
First, I create the users on a ceph admin vm:
radosgw-admin user create --tenant=homelab --uid=user1 --display-name="First User" ...
- 1
2
votes
1
answer
222
views
Where does CEPH store device metrics for each device? [ceph reef]
I just deployed my first ceph cluster and have been trying multiple things so far.
Though, I couldn't understand how ceph store metric for the devices.
I can easily access those through:
# ceph device ...
- 51
0
votes
1
answer
91
views
Completely Reprovisioning Ceph Rgw Without Reprovisioning Ceph Cluster
I recently began learning about Ceph and setup my own ansible scripts to setup a Ceph cluster (yes, I'm aware that cephadm-ansible exists, but I want to get comfortable managing ceph).
Initially, I ...
- 101
0
votes
0
answers
54
views
why rook cephfs storage class store pv data in 2 places?
I am a newbie to ceph and run ceph using rook in bare metal environment.my question is :
when you use rook cephfs provisioner as storage class and create a pvc to assign to a pod , when the pv is ...
- 101
1
vote
0
answers
292
views
Deleting file in ceph does not free up disk space
Ceph version: 16.2.13 (pacific) (I know that pacific is deprecated but the whole environment is legacy (like centos 7.3). I am not authorized to upgrade as well.) The cluster has 6 servers (22 osds, ...
0
votes
0
answers
45
views
Ceph rbd commands are hanging forever when executed on remote server
I have an openstack compute node connected to ceph cluster with 10G. The compute can ping and reach the all ceph nodes and port 3300 but somehow rbd commands are hanging which is stopping the compute ...
0
votes
1
answer
388
views
Ceph - reset ceph osd_max_backfills to default value
I am attempting to reset some backfill settings to defaults after having changed them to slow down the backfilling. I am struggling to work out how to do it though:
This example shows that the ...
0
votes
0
answers
59
views
openstack fails to create an instance after integrating with rbd ceph
I have configured openstack kolla to enable ceph for glance, cinder and nova. I must say that cinder and glance are working perfectly fine with ceph. Nova on the other hand refuses to work properly. ...
0
votes
1
answer
11k
views
Ceph hardware sizing calculator
I would like to calculate the Hardware sizing for a ceph cluster. There are so few references to the sizing, that I try to get these details here in the community.
E.g. what shall i have depending on
...
CommunityBot
- 1
0
votes
0
answers
48
views
openstack nova goes down when ceph pool is set to have more than 1 pgs
I have openstack connected to ceph cluster.
When i increase the pgs of the vms pool to be more than 1 and launch instance in openstack the instance is stuck on spawning state and nova compute ...
- 15
-1
votes
1
answer
389
views
Can I create buckets in ceph object storage using a command-line interface?
Is it possible to create buckets in the ceph object store via the rados gateway from a CLI? I am aware of the radosgw-admin tool, however, it doesn't appear to have this ability. It appears mainly to ...
- 19
0
votes
1
answer
1k
views
ceph "permission denied" when adding storage OSD
I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the ...
- 1
0
votes
1
answer
63
views
ceph snapshots are not running
Our ceph cluster (quincy 17.2.7) runs in a docker container. The snapshot schedule was created as:
ceph fs snap-schedule add / 1h 2023-12-13T09:00:00 nerfcomputing2024
and retention:
ceph fs snap-...
- 1
0
votes
0
answers
19
views
cephfs mount: can't open files with LibreOffice
I'm able to open/write different relevant filetypes with other applications (vim,code,gedit,...) from remote cephfs storage.
libreoffice has been configured to prevent file-locks.
All dirs and files ...
- 21
0
votes
0
answers
292
views
RADOS permission denied
I'm trying to install a ceph cluster using 3 ubuntu 20.04 machine each with 4 disk.
I've followed the manual installation from here https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-...
0
votes
1
answer
86
views
how number of replicas is aligned with crush rule in a ceph cluster
I have some difficulties to understand whether the crush rule is aligned with the number of replicas. So, imagine I create a crush rule named replicate_in_2_racks as follows:
ceph osd crush rule ...
- 25.1k
0
votes
2
answers
294
views
how to get the used space (size in bytes) of a directory in a ceph file system
I have a ceph cluster quincy 17.2.7
I wonder if ceph has some tools to quickly get the the used space (size in bytes) of a directory in a ceph file system. I know one can get that using du -hs /fs/dir-...
0
votes
0
answers
127
views
"ceph-volume lvm new-db" does not seem to perform as should be
A database device seems to be added to the OSD (via ceph-volume lvm new-db), despite an error is thrown.
Until now, it is unclear to me if the DB device was actually successfully added to the OSD or ...
- 1
0
votes
0
answers
401
views
Unable to mount CephFS on client - Rocky8.8
When attempting to mount a Ceph file system the mount command times out:
mount error 110 = Connection timed out
dmesg shows the following:
ceph: device name is missing path (no : separator in splunk@$...
- 101
0
votes
1
answer
52
views
ceph remove spurious data
I’d like to remove some “spurious" data:
root@XX03:/# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0
TOTAL ...
- 467
0
votes
0
answers
166
views
Ceph mounted rbd poor performance
all tests are done from an external server connected with 10G to all 3 ceph nodes and the cluster is all nvme flash.
when i ran fio with this config:
[global]
ioengine=rbd
clientname=admin
pool=...
0
votes
1
answer
329
views
Now that the in-tree Kubernetes CephFS persistent volume plugin is deprecated how can I use Rook to get the same mounting behavior?
As of Kubernetes v1.28 the CephFS in-tree persistent volume plugin is deprecated. The official suggestion is to now instead use the third-party CSI driver. However, I am not sure how I can use for ...
- 19
0
votes
0
answers
149
views
I'm trying to expose a folder size in Node exporter but messes up. any suggestions?
This wasn't the case for a few months but suddenly my Node exporter in the server service messed up.
Oct 02 10:13:01 nodefs01 node_exporter[117252]: time="2023-10-02T10:13:01Z" level=error ...
- 2,020
1
vote
1
answer
292
views
Kubernetes Rook-Ceph: different pools for different use-cases?
I'm experimenting with rook-ceph on kubernetes for storage again. I have 3 nodes, which each have
a local ssd-partition mounted at /dev/sda2, and
a NAS-volume at /dev/sdb.
In my rook-config, I use ...
0
votes
2
answers
95
views
how to use the images that have been create by "Offload rbd’s copy_volume_to_image" method in openstack?
I have an openstack integrated with ceph.
I'm working on the fastest way to create a copy of an instance's volume without using cinder_backup module.
in the Ceph it is possible to access the rbd files ...
- 3
0
votes
1
answer
160
views
extract openstack instances' volume from ceph
I have an OpenStack-kolla environment with the Ceph backend.
I've configured my cinder backend with NFS. I can create a Backup with Cinder from all instances and store them on NFS. In this case, I don'...
- 467
1
vote
1
answer
191
views
Ceph: Rados Gateway web endpoint not responding
I'm creating a simple Ceph cluster and trying to connect to Ceph Gateway.
This is ceph status output of my Ceph cluster:
cluster:
id: a7f64266-0894-4f1e-a635-d0aeaca0e993
health: ...
- 11
2
votes
2
answers
3k
views
Orchestrator is not available with fresh Rook instance
I'm trying to set up a rook Ceph cluster on my kubernetes cluster.
Topography:
3 kubernetes nodes (all are master/worker pods)
Each node has /dev/vdX on it for ceph
Each node is intended to work as ...
0
votes
1
answer
505
views
kolla-ansible openstack bootstrap-servers failed on enabling ceph apt repository (ubuntu 22.04)
This is to deploy the recent kolla-ansible openstack. The problem happens since last week. I retested it on a good environment successfully deployed on 7/14/2023 and got bootstrap-servers failed ...
- 1,194
0
votes
0
answers
59
views
Upgrade OS Bionic to Focal which includes Openstack and Ceph cluster
So i have openstack and ceph cluster with 3 controllers and 3 computes. This is the information about my cluster :
OS : ubuntu bionic
Openstack kolla-ansible : Ussuri 10.4.0
Ceph ceph-ansible : ...
4
votes
2
answers
2k
views
Ceph does not detect disks
I am using cephadm. I had a test cluster up and running. I re-installed the OS and during installation nuked the drives by deleting the LVM volume groups and partitions. Now lsblk shows the devices to ...
- 360
0
votes
0
answers
245
views
Ceph's failure to automount after network failure
I'm having some problems with the mounting of a ceph-cluster on debian machines, don't know if I'm doing something wrong, if it's a version problem or anything else.
I'm using the ceph cluster from ...
- 165
2
votes
2
answers
14k
views
MountVolume.MountDevice failed operation with the given Volume ID already exists
Environment:
Kubernetes cluster with 1 master and 3 nodes Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64) (VMWARE VMs)
screenshot from dashboard
Pod (simple nginx image) cannot be mounted ...
- 1
-1
votes
1
answer
585
views
Cephadm + Openstack Keystone Integration
Hello I am trying to Integrate Openstack Keystone as Ceph authentication Mechanism So i can use ceph object storage as openstack swift backend
Environment:
Kernel : Ubunutu Server LTS 22.04 (minimal)
...
- 467
0
votes
0
answers
136
views
ceph - is a drive mapping change recoverable /dev/sdi /dev/sdj
I was testing drive hot plugging on my server, and unplugged one of the drives, then plugged it back in. Though, when plugged back in the drive letter changed from /dev/sdj to /dev/sdl. Now ceph will ...
- 101
1
vote
0
answers
2k
views
Ceph reading and writing performance problems, fast reading and slow writing
Ceph reading and writing performance problems, fast reading and slow writing
Hello, we need to migrate all cloud environments to Proxmox. At present, I am evaluating and testing Proxmox+Ceph+OpenStack....
- 1
3
votes
1
answer
2k
views
Proxmox on Ceph performance & stability issues / Configuration doubts
We have just installed a cluster of 6 Proxmox servers, using 3 nodes as Ceph storage, and 3 nodes as compute nodes.
We are experiencing strange and critical issues with the performances and stability ...
- 203
9
votes
3
answers
8k
views
Is Ceph possible to handle hardware RAID arrays (LUNs) as OSD drives?
I am pretty new to Ceph and try to find out if Ceph supports hardware level raid HBAs.
Sadly could not find any information. What I found is, that it is recommended to use plain disks for OSD. But ...
- 103
0
votes
0
answers
52
views
Ceph File system OS installation
I'm building a Ceph cluster.
I've been reading for months and I can't find this information.
I am going to use Ubuntu 20.04
What is the file system that I have to use for the installation of the ...
- 36.8k
3
votes
1
answer
137
views
Pooled storage with varying redundancy per file system
I have some files that I want stored mirrored. I have some files that I only need single copies (i.e. scratch data, easily regenerated data, etc). And I have some files that are so critical, I want it ...
- 50.7k
0
votes
1
answer
77
views
OSD stuck in booting with messenger component log: Operation not permitted error
After rebooting k8s nodes, OSD didn't join the cluster with errors related to authentication. I have added them to auth list and that error disappears.
Now OSD nodes join the cluster but they don't ...
1
vote
0
answers
155
views
Error deploying Ceph Quincy using ceph-ansible 7 on Rocky 9
I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am getting some problems and I don't know where to search for the reason.
PS : I did the same deployment on Rocky8 using ceph-ansible ...
- 712
0
votes
1
answer
1k
views
Why is ceph is not detecting ssd device on a new node?
I have installed a ceph cluster (quincy) with already 2 nodes and 4 OSDs.
Now I added a 3rd host running Debian (bullseye) to the cluster. The new host is deteced correctly and runs a mom.
The problem ...
- 21
0
votes
1
answer
564
views
Ceph determine PG based on rados object name
In Ceph, the command rados --pgid <pgid> ls allows you to list all the rados objects that are stored in a specific placement group. My question is: is there a command that does the reverse? So, ...
- 467
0
votes
2
answers
2k
views
Which Ceph OSD is mapped to which pysical drive
I have a Ceph system with 8 OSD's and 8 disks mapped 1:1.
One of the disks is giving me smart errors and I would like to replace it.
How do I know which physical disk is mapped to which OSD?
- 467