Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
150
questions
0
votes
0
answers
19
views
Erasure code overhead for heterogeneous hardware
I need to setup a Ceph cluster on a set of 10 machines having different disk sizes and numbers. The numbers ranges from 16x500GB to 8x4TB, with various intermediate values, the most common being 8x2TB....
- 261
0
votes
0
answers
18
views
Is there a way to check filesystem/files on a logical drive in CEPH storage environment
Little background.
HP C7K hardware
Example partition:
/dev/sdi :
/dev/sdi1 ceph data, prepared, cluster ceph, osd.21, block /dev/sdi2, block.db /dev/nvme1n1p3, block.wal /dev/nvme1n1p4
/dev/sdi2 ...
- 1
0
votes
1
answer
91
views
Remove orphaned Ceph volumes
I have installed a Ceph cluster across three nodes which needed completely reinstalling after testing. It seems that on one of the nodes, some configuration data remained which Ceph is still picking ...
- 1,269
0
votes
0
answers
58
views
Usage stats of CephFS subvolumes
Is there a way to gather and show the disk usage stats of CephFS subvolumes?
Is there anything similar to rbd du for CephFS subvolumes?
The only idea I have is something really ugly:
filesystem="...
- 537
1
vote
0
answers
47
views
Computing stripe count for erasure coded storage
I'm setting up a ceph cluster (first time for me) which in the end will be made of ~100 disks spread over 10 hosts. I'm going with a single erasure coded data pool to maximize disk space; my ...
- 261
0
votes
1
answer
70
views
Ceph RGW 16.2.7 CLI changes
I am trying to run a new Ceph cluster with Rados GW using the last software version 16.2.7, but when I set up RGW nodes I found out there are some changes in the CLI comparing with a version 16.2.4 I ...
- 578
0
votes
0
answers
87
views
Openstack Glance Image Corrupted
I have an openstack instance up and running for 4 months and would like to migrate it to another nova node. When I click migrate instance option of that 4 month running instance on openstack dashboard....
- 33
0
votes
0
answers
80
views
Deploying Ceph with RGW Ingress uses [::]:Port instead of virtual ip
I'm deploying regular Ceph Pacific using cephadm, including RGW.
Now I want to use the ingress-service, which includes specifying a virtual ip - so far so good.
Unfortunately this service now creates ...
0
votes
2
answers
1k
views
ssh-copy-id ceph@osd-0 /bin/ssh-copy-id: ERROR: failed to open ID file '/home/ceph/.pub': No such file or directory
I can ssh successfully my ceph osd nodes but when i am trying to copy ssh id i am getting below error can anyone guide why i am getting below errors.
[ceph@monitor ~]$ ssh-copy-id ceph@osd-0
/bin/ssh-...
- 11
2
votes
0
answers
674
views
Proxmox on Ceph performance & stability issues / Configuration doubts
We have just installed a cluster of 6 Proxmox servers, using 3 nodes as Ceph storage, and 3 nodes as compute nodes.
We are experiencing strange and critical issues with the performances and stability ...
- 163
1
vote
0
answers
102
views
Ceph RGW: slow `list_bucket` requests
I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds:
$ ...
- 1,922
0
votes
1
answer
45
views
Changing Ceph to store number of data copies
Currently I am using replication for data placement and think I am using three copies, which I think is the default. How do I change ceph config to store 4 copies on different nodes in different ...
0
votes
1
answer
110
views
Three Single drive Raid 1 vs JBOD
I have an intel server of which the RAID Controller does not support JBOD mode (same problem as my Dell R820s).
But I want to do Ceph, so I "need" a JBOD. So I set my 3 SSDs as single drive ...
- 101
3
votes
0
answers
127
views
Openstack Nova and Ceph Volume Attachment Issue
I am trying out the volume attachment function at Openstack (version:wallaby) to the server as additional device but failed.
The volume backend is ceph which all of the osds are up and healthy.
ceph-...
- 33
0
votes
1
answer
42
views
Openstack Wallaby Volume Backup Issue
Recently I am testing kubernetes with 1 master node and 3 worker nodes that built on Openstack(version: Wallaby). It involves lots of testing and therefore I would like to back up the volume to local ...
- 33
0
votes
1
answer
48
views
Ceph connect to local node
I have an idea for an application that I'd like to build and one of the requirements is a globally replicated filesystem. Things like Ceph and GlusterFS exist, but I'm not sure they meet my particular ...
- 135
0
votes
1
answer
397
views
cannot Delete objects on s3 ceph bucket
I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
0
votes
0
answers
50
views
Reduced ceph storage pool size
Currently running a 7-node Ceph cluster, used as file system storage.
Two independent nodes are used to run the mon+mgr+mds service, the other 5 nodes are used as storage nodes and one of the storage ...
3
votes
0
answers
189
views
How to create Ceph OSDs on Partitions using cephadm, on ceph-pacific?
In the pacific version of Ceph, OSD creation is not allowing by cephadm on partitions.(We have NvMes with partitions). The command used is:
ceph orch daemon add osd <host>:<device-path>
...
- 41
0
votes
0
answers
105
views
nfs (or ceph, or k8s) works strange: ls/tree not working, but i can read files
I have some problems and i want to know where i have to check.
my env: ceph nfs mount to a k8s pod
problem:
at mounted path (on k8s node and inside the pod too), ls/tree not working, but i can read/...
- 1
0
votes
1
answer
306
views
Multiple port on the same server behind the same backend
I’m using haproxy with my ceph cluster and I’ve created more gateways on 1 server with different ports. I’ve added to the backend configuration with the different port numbers but seems like haproxy ...
- 15
0
votes
1
answer
442
views
How can I extend or resize lvm from disk space
We use ceph to manage the storage and recently increased /dev/sdb's disk size.
I'm not sure how to utilize the newly available space in my ceph's lvm.
sdb ...
- 1
2
votes
0
answers
74
views
CEPH/ RGW encryption with mutliple tenants
I'm currently planning a small OpenStack deployment and I want to use Ceph for object storage (via Rados Gateway) and block storage.
Ceph supports multitenancy. It also supports encryption using key ...
1
vote
1
answer
736
views
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted
When I have upgrade the from Octopus 15.2.10 to Pacific 16.2.0, the mon nodes starts successfully using the manual upgrade process (by installing the packages with no orch)
however when I have upgrade ...
- 27
2
votes
1
answer
719
views
Orchestrator is not available with fresh Rook instance
I'm trying to set up a rook Ceph cluster on my kubernetes cluster.
Topography:
3 kubernetes nodes (all are master/worker pods)
Each node has /dev/vdX on it for ceph
Each node is intended to work as ...
- 613
0
votes
1
answer
305
views
Multi site ceph + proxmox
We have 2 proxmox clusters in production.
They contain Ceph cluster with cephfs is on it.
The task is to sync files between sites. Site A is master and site B is for backup.
On site A cephfs files are ...
0
votes
1
answer
353
views
How to know how many files can Ceph delete per 1 minute?
As the question says. Is there a way to find out how many files can Ceph delete per 1 minute?
A little backstory: Image hosting website, running Ceph FS on 8TB WD RED 3.5 HDD's, around 51 TiB already ...
- 1
0
votes
0
answers
801
views
Cinder volume service goes down while creating/deleting multiple volumes at the same time
Environment:
Openstack Rocky
Ceph Mimic
openstack-cinder-13.0.8-0
Cinder volume status goes down while creating or deleting multiple volumes at the same time manually or while Rally test is running.
...
0
votes
1
answer
1k
views
Ceph Cluster - Reduced data availability: 96 pgs inactive And All OSD Nodes Are Down
I set up my Ceph Cluster by following this document. I have one Manager Node, one Monitor Node, and three OSD Nodes. The problem is that right after I finished setting up the cluster, the ceph health ...
0
votes
1
answer
426
views
Pipe rbd export comand to tar to create compressed export
Is it possible to pipe the output of rbd export directly to something else like tar to create a compressed archive of the .img file without having to write to disk first?
Something like
rbd export ...
- 613
1
vote
1
answer
796
views
Ceph: is setting lower "size" parameter on a live pool possible?
I have a 5-node Proxmox cluster using Ceph as the primary VM storage backend. The Ceph pool is currently configured with a size of 5 (1 data replica per OSD per node) and a min_size of 1. Due to the ...
- 13
0
votes
1
answer
134
views
Openstack Using Ceph Erasure Code Pool
I tried configuring openstack with ceph pool erasure code backend
I'm trying to using —data-pool on ceph.conf using rbd default data pool like the following blog https://themeanti.me/technology/2018/...
1
vote
1
answer
236
views
Recommended values for mon osd down out in ceph
Are there any recommended values for "mon osd down out"?
The cluster has 15 nodes and 300 osds. It's running ceph Jewl.
The default value in Jewel is 300 seconds. I was wondering, are there ...
0
votes
2
answers
1k
views
Proxmox Ceph - Got timeout on separate network
I've installed on 4 nodes a completly fresh OS with Proxmox. Every node has 2xNVMe und 1xHD, one NIC public, one NIC private. On the public network there is an additional wireguard interface running ...
- 408
1
vote
0
answers
245
views
how to sync ceph bucket with scaleway s3 bucket or any s3 bucket for?
i,m trying to sync a ceph cluster to a scaleway s3 bucket
i have the secret, access token and https endpoint for my s3 bucket
ceph version: mimic
i followed the cloud sync module guide at the ...
- 83
1
vote
0
answers
61
views
Ceph-ansible: using the OS-SSD as cache
In ceph-ansible, you can specify a disk to be used for databases in the hosts file using the dedicated_devices directory. Example:
hosts:
stor01.hostname.com
devices: ['/dev/sdd', '/dev/...
- 111
3
votes
1
answer
3k
views
Ceph e5 handle_auth_request failed to assign global_id after a host outage
I have a small 3-host Ceph cluster with Ubuntu 20.04.1 and Ceph 15.2.5 using docker containers and deployed with cephadm. Yesterday one of the hosts (s65-ceph) had a power outage. The other two ...
- 71
0
votes
1
answer
118
views
ceph-deploy install --release fails to upgrade on debian stretch
this is my first post on serverfault!
I'm trying to setup a test ceph cluster to see if it can meet our storage needs.
I've been following the instructions here
I ran into some permissions issues when ...
- 1
0
votes
3
answers
737
views
Data resiliency in Ceph or Gluster
Say I am most concerned with keeping my data safe from physical hard drive failure and the computer it connects to directly.
RAID1 and RAID5 I understand. They provide protections against data loss ...
- 41
0
votes
1
answer
203
views
Memory level for Ceph logs. What's this for?
I want to increase the logging level of some OSDs, but I don't understand what the memory level is for. I didn't find an exact explanation in the documentation. Is there a need to increase this level? ...
- 191
1
vote
0
answers
1k
views
CEPH cluster became unresponsive: e5 handle_auth_request failed to assign global_id
I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 hosts in total, all ESXi VMs). It lived through a couple of reboots without problem, then I've reconfigured the main host a bit:
...
- 65
1
vote
1
answer
88
views
Can my Ceph nodes have the same hostname?
I have a high availability solution which requires me to have all my nodes using the same hostname. I've been tasked to use ceph cluster storage, but I just reached a dead end.
I can't figure out how ...
- 21
3
votes
2
answers
704
views
How to best use large NVMe array for VMWare Datastore
I recently acquired a Supermicro SYS-1029P-N32R with 16 14TB NVMe SSD's (Micron 9300 Pro) at work. We have 100 gigabit networking on the box to our VMWare hosts. We've tried using FreeNAS to host ZFS ...
3
votes
0
answers
86
views
Ceph Rados GW Proxy
I've been given an access key to a Ceph cluster that runs radoswgw to provide S3. The key allows bucket creation, object reading etc. I don't admin the Ceph or radosgw.
I have a group of users that ...
- 131
0
votes
1
answer
595
views
Does Kubeadm reset delete persistent volumes
I need to change the advertise address of my Kubernetes master node.
I can do this with a kubeadm reset, but I don't want to lose my (rook-ceph) persistent volumes on the master node.
Is it safe to do ...
0
votes
1
answer
59
views
OpenStack redundant shared storage backend solution
Which storage backends are recommended for OpenStack which allows live migration and shared storage?
Is it NFS / Ceph / storage backend drivers to commercial storage solutions? Which of these is ...
- 98
0
votes
1
answer
380
views
Error running ceph-deploy install
I have been following the following tutorial to setup ceph on CentOS
https://wiki.centos.org/SpecialInterestGroup/Storage/ceph-Quickstart
When I try to run ceph-deploy to install to my other nodes, ...
- 2,633
-2
votes
1
answer
71
views
Backup solution for medium infrastructure with commodity hardware
I'm looking for recommendations for a backup solution for our Infrastructure.
We have lots of old Hardware, which is not very fast but just laying around.
We have like 20 Servers for backup with 8-...
1
vote
1
answer
2k
views
Poor performance with rook, ceph and RBD
I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD.
The OSDs are using the same disk as the VM Operating System.
The VM disks are remote (...
0
votes
1
answer
6k
views
Ceph hardware sizing calculator
I would like to calculate the Hardware sizing for a ceph cluster. There are so few references to the sizing, that I try to get these details here in the community.
E.g. what shall i have depending on
...
- 277