Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
173
questions
0
votes
0
answers
19
views
ceph "permission denied" when adding storage OSD
I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the ...
2
votes
1
answer
25
views
Pooled storage with varying redundancy per file system
I have some files that I want stored mirrored. I have some files that I only need single copies (i.e. scratch data, easily regenerated data, etc). And I have some files that are so critical, I want it ...
- 204
0
votes
1
answer
19
views
OSD stuck in booting with messenger component log: Operation not permitted error
After rebooting k8s nodes, OSD didn't join the cluster with errors related to authentication. I have added them to auth list and that error disappears.
Now OSD nodes join the cluster but they don't ...
- 103
0
votes
0
answers
73
views
Ceph - reset ceph osd_max_backfills to default value
I am attempting to reset some backfill settings to defaults after having changed them to slow down the backfilling. I am struggling to work out how to do it though:
This example shows that the ...
- 101
0
votes
1
answer
82
views
Ceph determine PG based on rados object name
In Ceph, the command rados --pgid <pgid> ls allows you to list all the rados objects that are stored in a specific placement group. My question is: is there a command that does the reverse? So, ...
- 1,754
1
vote
0
answers
60
views
Error deploying Ceph Quincy using ceph-ansible 7 on Rocky 9
I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am getting some problems and I don't know where to search for the reason.
PS : I did the same deployment on Rocky8 using ceph-ansible ...
- 51
0
votes
0
answers
168
views
changing queue parameter nr_request on NVMe dev
We would like to use NVMes (SAMSUNG MZ1LB1T9HALS-00007) as a cache-tier in a Ceph System (17.2.0). The problem is that the stored data has a lot of small files and if hot-tiering is set to ~70% on the ...
- 101
0
votes
2
answers
215
views
Which Ceph OSD is mapped to which pysical drive
I have a Ceph system with 8 OSD's and 8 disks mapped 1:1.
One of the disks is giving me smart errors and I would like to replace it.
How do I know which physical disk is mapped to which OSD?
- 128
0
votes
1
answer
70
views
ceph-deploy install command fails with [ceph_deploy][ERROR ] RuntimeError: configparser.NoSectionError: No section: 'main'
Command ceph-deploy install admin datanode_dn2 fails with output:
[ceph_deploy.install][INFO ] Distro info: rocky 9.1 blue onyx
[admin][INFO ] installing Ceph on admin
[admin][INFO ] Running ...
- 133
0
votes
1
answer
375
views
Ceph does not detect disks
I am using cephadm. I had a test cluster up and running. I re-installed the OS and during installation nuked the drives by deleting the LVM volume groups and partitions. Now lsblk shows the devices to ...
- 101
0
votes
0
answers
535
views
Ceph reading and writing performance problems, fast reading and slow writing
Ceph reading and writing performance problems, fast reading and slow writing
Hello, we need to migrate all cloud environments to Proxmox. At present, I am evaluating and testing Proxmox+Ceph+OpenStack....
1
vote
2
answers
78
views
Configure ceph block storage pool in cockpit
I have a plain install on machine C with Oracle Linux 9.1 and cockpit and cockpit-machines. On machines A and B I have a ceph cluster configured which defines an rbd block storage pool for VM disks. ...
- 121
0
votes
1
answer
89
views
Cache mechanisme in kubernetes
I've got the next setup:
Proxmox 7.2
CEPH 16.2.9
K3S v1.23.15+k3s1
CEPH CSI v3.7.2
CEPH using as RBD-storage for QEMU images and K8S PVC. When I do disk benchmark in QEMU I've got the next results:
...
- 33
-1
votes
1
answer
281
views
Cephadm + Openstack Keystone Integration
Hello I am trying to Integrate Openstack Keystone as Ceph authentication Mechanism So i can use ceph object storage as openstack swift backend
Environment:
Kernel : Ubunutu Server LTS 22.04 (minimal)
...
- 109
0
votes
1
answer
396
views
Why is ceph is not detecting ssd device on a new node?
I have installed a ceph cluster (quincy) with already 2 nodes and 4 OSDs.
Now I added a 3rd host running Debian (bullseye) to the cluster. The new host is deteced correctly and runs a mom.
The problem ...
- 101
0
votes
1
answer
60
views
Enabling Cephx In cephadm depolyment
Hello i am trying to enable cephx in my ceph cluster
i added this three lines to /etc/ceph/ceph.conf
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
i didn'...
- 109
0
votes
0
answers
298
views
Unable to start Ceph RGW service
Context: I'm trying to setup a new ceph-17.2.5 cluster Initial setup consists of 1 mgr, 1 mon and 1 osd all installed on same box using cephadm.
So far I am able to access http://192.168.3.50:8080/api/...
1
vote
1
answer
306
views
Ceph alertmanager config
I've installed ceph using cephadm
Including monitoring stack prometheus, alertmanager, node-exporter
Currently I'm trying to add a telegram receiver (Telegram is supported from v0.24.0, so I've ...
- 63
1
vote
0
answers
25
views
Ceph reports bucket space utilization and total cluster utilization that is inconsistent
I copied the contents of an older Ceph cluster to a new Ceph cluster using rclone. Because several of the buckets had tens of millions of objects in a single directory I had to enumerate these ...
- 111
0
votes
1
answer
63
views
In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules? [duplicate]
I'm designing a ceph cluster for mixing cephFS and rbd for my company( VM et file infrastructure).
In my set I need 32To raw storage by node. I start with 5 nodes.
The seller quotes propose me to ...
0
votes
0
answers
64
views
Dedicated Network connection for CEPH or aggregated general purpose network connection?
We are building a Kubernetes Cluster on bare metal. As the cluster is getting physically installed, I was asked to weigh in on the following question:
Each node has two network connections of 25Gbps. ...
0
votes
0
answers
75
views
Where do I find a script to check a Ceph iSCSI gateway configuration?
On a Ceph cluster with iSCSI gateway we have an issue with gateway configuration changes being applied partially. The result is a broken gateway.conf, which goes unnoticed until (possibly weeks later) ...
- 411
1
vote
0
answers
516
views
How to configure a CEPH host with public and private NICs
So here is the situation. I am deploying CEPH on a public cloud using cephadm.
I am trying to create a private,public network config as described here
First we bootstrap the monitor with
ceph ...
- 11
0
votes
1
answer
123
views
Is running CEPH on public cloud without data replication a good idea?
My startup is thinking about adopting CEPH for block storage (and other features it provides). Most of our infrastructure runs on public cloud (DO,GCP,AWS,etc...).
On public cloud data storage you ...
- 11
0
votes
0
answers
50
views
Serve cephFS files over HTTP (or S3?)
I have a CEPH+LXD cluster configuration.
And I would like to serve the files in the container over HTTP (S3,whatever doesn't matter).
Is there a way to do this using the rgw service?
Like can I point ...
- 11
0
votes
0
answers
118
views
Volume not getting mounted (Nexus is running on this Pod)
Nexus is running on worker node and Nexus UI went down because of storage issue. We increased the storage space from 150 GB to 210GB for PV & PVC.
Increased storage for PV and PVC
Then, when we ...
- 1
0
votes
0
answers
230
views
Ceph : Deleted bucket still exist
I have tested multiple configuration of pool and created multiple placement target.
However i had deleted placement group and pools (index, non-ec and data pool) before deleting the bucket.
Since this ...
- 155
0
votes
1
answer
640
views
Remove orphaned Ceph volumes
I have installed a Ceph cluster across three nodes which needed completely reinstalling after testing. It seems that on one of the nodes, some configuration data remained which Ceph is still picking ...
- 1,289
0
votes
0
answers
234
views
Usage stats of CephFS subvolumes
Is there a way to gather and show the disk usage stats of CephFS subvolumes?
Is there anything similar to rbd du for CephFS subvolumes?
The only idea I have is something really ugly:
filesystem="...
- 577
1
vote
0
answers
111
views
Computing stripe count for erasure coded storage
I'm setting up a ceph cluster (first time for me) which in the end will be made of ~100 disks spread over 10 hosts. I'm going with a single erasure coded data pool to maximize disk space; my ...
- 271
0
votes
1
answer
249
views
Ceph RGW 16.2.7 CLI changes
I am trying to run a new Ceph cluster with Rados GW using the last software version 16.2.7, but when I set up RGW nodes I found out there are some changes in the CLI comparing with a version 16.2.4 I ...
- 578
0
votes
0
answers
324
views
Openstack Glance Image Corrupted
I have an openstack instance up and running for 4 months and would like to migrate it to another nova node. When I click migrate instance option of that 4 month running instance on openstack dashboard....
- 33
0
votes
2
answers
9k
views
ssh-copy-id ceph@osd-0 /bin/ssh-copy-id: ERROR: failed to open ID file '/home/ceph/.pub': No such file or directory
I can ssh successfully my ceph osd nodes but when i am trying to copy ssh id i am getting below error can anyone guide why i am getting below errors.
[ceph@monitor ~]$ ssh-copy-id ceph@osd-0
/bin/ssh-...
- 11
2
votes
0
answers
2k
views
Proxmox on Ceph performance & stability issues / Configuration doubts
We have just installed a cluster of 6 Proxmox servers, using 3 nodes as Ceph storage, and 3 nodes as compute nodes.
We are experiencing strange and critical issues with the performances and stability ...
- 163
1
vote
0
answers
301
views
Ceph RGW: slow `list_bucket` requests
I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds:
$ ...
- 2,102
0
votes
1
answer
281
views
Changing Ceph to store number of data copies
Currently I am using replication for data placement and think I am using three copies, which I think is the default. How do I change ceph config to store 4 copies on different nodes in different ...
0
votes
1
answer
308
views
Three Single drive Raid 1 vs JBOD
I have an intel server of which the RAID Controller does not support JBOD mode (same problem as my Dell R820s).
But I want to do Ceph, so I "need" a JBOD. So I set my 3 SSDs as single drive ...
- 101
3
votes
0
answers
378
views
Openstack Nova and Ceph Volume Attachment Issue
I am trying out the volume attachment function at Openstack (version:wallaby) to the server as additional device but failed.
The volume backend is ceph which all of the osds are up and healthy.
ceph-...
- 33
0
votes
1
answer
72
views
Openstack Wallaby Volume Backup Issue
Recently I am testing kubernetes with 1 master node and 3 worker nodes that built on Openstack(version: Wallaby). It involves lots of testing and therefore I would like to back up the volume to local ...
- 33
0
votes
1
answer
146
views
Ceph connect to local node
I have an idea for an application that I'd like to build and one of the requirements is a globally replicated filesystem. Things like Ceph and GlusterFS exist, but I'm not sure they meet my particular ...
- 135
0
votes
1
answer
682
views
cannot Delete objects on s3 ceph bucket
I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
3
votes
0
answers
449
views
How to create Ceph OSDs on Partitions using cephadm, on ceph-pacific?
In the pacific version of Ceph, OSD creation is not allowing by cephadm on partitions.(We have NvMes with partitions). The command used is:
ceph orch daemon add osd <host>:<device-path>
...
- 41
0
votes
0
answers
192
views
nfs (or ceph, or k8s) works strange: ls/tree not working, but i can read files
I have some problems and i want to know where i have to check.
my env: ceph nfs mount to a k8s pod
problem:
at mounted path (on k8s node and inside the pod too), ls/tree not working, but i can read/...
- 1
0
votes
1
answer
881
views
Multiple port on the same server behind the same backend
I’m using haproxy with my ceph cluster and I’ve created more gateways on 1 server with different ports. I’ve added to the backend configuration with the different port numbers but seems like haproxy ...
- 115
0
votes
1
answer
803
views
How can I extend or resize lvm from disk space
We use ceph to manage the storage and recently increased /dev/sdb's disk size.
I'm not sure how to utilize the newly available space in my ceph's lvm.
sdb ...
- 1
2
votes
0
answers
100
views
CEPH/ RGW encryption with mutliple tenants
I'm currently planning a small OpenStack deployment and I want to use Ceph for object storage (via Rados Gateway) and block storage.
Ceph supports multitenancy. It also supports encryption using key ...
2
votes
1
answer
1k
views
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted
When I have upgrade the from Octopus 15.2.10 to Pacific 16.2.0, the mon nodes starts successfully using the manual upgrade process (by installing the packages with no orch)
however when I have upgrade ...
- 37
2
votes
1
answer
2k
views
Orchestrator is not available with fresh Rook instance
I'm trying to set up a rook Ceph cluster on my kubernetes cluster.
Topography:
3 kubernetes nodes (all are master/worker pods)
Each node has /dev/vdX on it for ceph
Each node is intended to work as ...
- 593
0
votes
1
answer
434
views
Multi site ceph + proxmox
We have 2 proxmox clusters in production.
They contain Ceph cluster with cephfs is on it.
The task is to sync files between sites. Site A is master and site B is for backup.
On site A cephfs files are ...
0
votes
1
answer
732
views
How to know how many files can Ceph delete per 1 minute?
As the question says. Is there a way to find out how many files can Ceph delete per 1 minute?
A little backstory: Image hosting website, running Ceph FS on 8TB WD RED 3.5 HDD's, around 51 TiB already ...
- 1