Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
133
questions
0
votes
1answer
27 views
How can I extend or resize lvm from disk space
We use ceph to manage the storage and recently increased /dev/sdb's disk size.
I'm not sure how to utilize the newly available space in my ceph's lvm.
sdb ...
2
votes
0answers
33 views
CEPH/ RGW encryption with mutliple tenants
I'm currently planning a small OpenStack deployment and I want to use Ceph for object storage (via Rados Gateway) and block storage.
Ceph supports multitenancy. It also supports encryption using key ...
1
vote
1answer
250 views
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted
When I have upgrade the from Octopus 15.2.10 to Pacific 16.2.0, the mon nodes starts successfully using the manual upgrade process (by installing the packages with no orch)
however when I have upgrade ...
0
votes
0answers
21 views
Ceph mon ms_dispatch segmentation fault
I have faced the following error message as I try to start the monitoring node (there is only 1 monitoring node)
any help is much appreciated.
I am running Ceph on Ubuntu 20.04 with latest update
2
votes
1answer
93 views
Orchestrator is not available with fresh Rook instance
I'm trying to set up a rook Ceph cluster on my kubernetes cluster.
Topography:
3 kubernetes nodes (all are master/worker pods)
Each node has /dev/vdX on it for ceph
Each node is intended to work as ...
0
votes
1answer
100 views
Multi site ceph + proxmox
We have 2 proxmox clusters in production.
They contain Ceph cluster with cephfs is on it.
The task is to sync files between sites. Site A is master and site B is for backup.
On site A cephfs files are ...
0
votes
1answer
46 views
How to know how many files can Ceph delete per 1 minute?
As the question says. Is there a way to find out how many files can Ceph delete per 1 minute?
A little backstory: Image hosting website, running Ceph FS on 8TB WD RED 3.5 HDD's, around 51 TiB already ...
0
votes
0answers
136 views
Cinder volume service goes down while creating/deleting multiple volumes at the same time
Environment:
Openstack Rocky
Ceph Mimic
openstack-cinder-13.0.8-0
Cinder volume status goes down while creating or deleting multiple volumes at the same time manually or while Rally test is running.
...
0
votes
1answer
351 views
Ceph Cluster - Reduced data availability: 96 pgs inactive And All OSD Nodes Are Down
I set up my Ceph Cluster by following this document. I have one Manager Node, one Monitor Node, and three OSD Nodes. The problem is that right after I finished setting up the cluster, the ceph health ...
0
votes
0answers
131 views
Create a fio job with multiple read and writes
I am trying to create a fio job that:
will write 10% of size with verify=crc32c (to save the hash for next steps).
how to set the location where the hash files are stored?
After step one,
there ...
0
votes
1answer
83 views
Pipe rbd export comand to tar to create compressed export
Is it possible to pipe the output of rbd export directly to something else like tar to create a compressed archive of the .img file without having to write to disk first?
Something like
rbd export ...
1
vote
1answer
181 views
Ceph: is setting lower “size” parameter on a live pool possible?
I have a 5-node Proxmox cluster using Ceph as the primary VM storage backend. The Ceph pool is currently configured with a size of 5 (1 data replica per OSD per node) and a min_size of 1. Due to the ...
0
votes
1answer
53 views
Openstack Using Ceph Erasure Code Pool
I tried configuring openstack with ceph pool erasure code backend
I'm trying to using —data-pool on ceph.conf using rbd default data pool like the following blog https://themeanti.me/technology/2018/...
1
vote
1answer
61 views
Recommended values for mon osd down out in ceph
Are there any recommended values for "mon osd down out"?
The cluster has 15 nodes and 300 osds. It's running ceph Jewl.
The default value in Jewel is 300 seconds. I was wondering, are there ...
0
votes
1answer
401 views
Proxmox Ceph - Got timeout on separate network
I've installed on 4 nodes a completly fresh OS with Proxmox. Every node has 2xNVMe und 1xHD, one NIC public, one NIC private. On the public network there is an additional wireguard interface running ...
1
vote
0answers
110 views
how to sync ceph bucket with scaleway s3 bucket or any s3 bucket for?
i,m trying to sync a ceph cluster to a scaleway s3 bucket
i have the secret, access token and https endpoint for my s3 bucket
ceph version: mimic
i followed the cloud sync module guide at the ...
1
vote
0answers
21 views
Ceph-ansible: using the OS-SSD as cache
In ceph-ansible, you can specify a disk to be used for databases in the hosts file using the dedicated_devices directory. Example:
hosts:
stor01.hostname.com
devices: ['/dev/sdd', '/dev/...
0
votes
0answers
113 views
ensure cinder-api is available timeout
I can't install a new openstack-ansible (with ceph-ansible) cloud. The "setup-openstack" playbook gets an timeout at the task: os_cinder : Ensure cinder api is available timeout. It doesn't ...
1
vote
1answer
1k views
Ceph e5 handle_auth_request failed to assign global_id after a host outage
I have a small 3-host Ceph cluster with Ubuntu 20.04.1 and Ceph 15.2.5 using docker containers and deployed with cephadm. Yesterday one of the hosts (s65-ceph) had a power outage. The other two ...
0
votes
0answers
54 views
cephfs path on samba cluster not found
I have a ceph cluster running a cephfs. I also have a samba cluster (ctdb) and want to share different folders (lets say projects and plm) on the cephfs. But I can only share the root directory of the ...
0
votes
1answer
69 views
ceph-deploy install --release fails to upgrade on debian stretch
this is my first post on serverfault!
I'm trying to setup a test ceph cluster to see if it can meet our storage needs.
I've been following the instructions here
I ran into some permissions issues when ...
0
votes
3answers
390 views
Data resiliency in Ceph or Gluster
Say I am most concerned with keeping my data safe from physical hard drive failure and the computer it connects to directly.
RAID1 and RAID5 I understand. They provide protections against data loss ...
0
votes
1answer
104 views
Memory level for Ceph logs. What's this for?
I want to increase the logging level of some OSDs, but I don't understand what the memory level is for. I didn't find an exact explanation in the documentation. Is there a need to increase this level? ...
0
votes
0answers
814 views
CEPH cluster became unresponsive: e5 handle_auth_request failed to assign global_id
I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 hosts in total, all ESXi VMs). It lived through a couple of reboots without problem, then I've reconfigured the main host a bit:
...
1
vote
1answer
51 views
Can my Ceph nodes have the same hostname?
I have a high availability solution which requires me to have all my nodes using the same hostname. I've been tasked to use ceph cluster storage, but I just reached a dead end.
I can't figure out how ...
3
votes
2answers
505 views
How to best use large NVMe array for VMWare Datastore
I recently acquired a Supermicro SYS-1029P-N32R with 16 14TB NVMe SSD's (Micron 9300 Pro) at work. We have 100 gigabit networking on the box to our VMWare hosts. We've tried using FreeNAS to host ZFS ...
3
votes
0answers
50 views
Ceph Rados GW Proxy
I've been given an access key to a Ceph cluster that runs radoswgw to provide S3. The key allows bucket creation, object reading etc. I don't admin the Ceph or radosgw.
I have a group of users that ...
0
votes
1answer
206 views
Does Kubeadm reset delete persistent volumes
I need to change the advertise address of my Kubernetes master node.
I can do this with a kubeadm reset, but I don't want to lose my (rook-ceph) persistent volumes on the master node.
Is it safe to do ...
0
votes
1answer
47 views
OpenStack redundant shared storage backend solution
Which storage backends are recommended for OpenStack which allows live migration and shared storage?
Is it NFS / Ceph / storage backend drivers to commercial storage solutions? Which of these is ...
0
votes
1answer
215 views
Error running ceph-deploy install
I have been following the following tutorial to setup ceph on CentOS
https://wiki.centos.org/SpecialInterestGroup/Storage/ceph-Quickstart
When I try to run ceph-deploy to install to my other nodes, ...
-2
votes
1answer
57 views
Backup solution for medium infrastructure with commodity hardware
I'm looking for recommendations for a backup solution for our Infrastructure.
We have lots of old Hardware, which is not very fast but just laying around.
We have like 20 Servers for backup with 8-...
1
vote
1answer
1k views
Poor performance with rook, ceph and RBD
I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD.
The OSDs are using the same disk as the VM Operating System.
The VM disks are remote (...
0
votes
1answer
4k views
Ceph hardware sizing calculator
I would like to calculate the Hardware sizing for a ceph cluster. There are so few references to the sizing, that I try to get these details here in the community.
E.g. what shall i have depending on
...
1
vote
1answer
105 views
Ceph with mainstream (desktop) SSDs?
I am searching for an answer about using SSDs drivers, which are build for the mainstream.
Afaik ceph uses for its marketing to buy cheap disks for entprise ready storages. But I cannot find any ...
1
vote
0answers
48 views
Ceph Clock Skew
I want to know where I can configure the limit of skew time for my ceph monitors. Also,how does CEPH throw this clock skew error? Specifically from which file and where can I find the file so that I ...
1
vote
0answers
52 views
CEPH netfilter connection tracking issues
We have a CEPH cluster (Ubuntu 18.04, Luminous) for Openstack images and volumes. As I was taking it into production I found many performance issues, slow OSDs, and throughput down to a trickle; this ...
1
vote
0answers
3k views
MountVolume.MountDevice failed operation with the given Volume ID already exists
Environment:
Kubernetes cluster with 1 master and 3 nodes Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64) (VMWARE VMs)
screenshot from dashboard
Pod (simple nginx image) cannot be mounted ...
2
votes
1answer
188 views
Setup ceph for storage mailserver?
I have read some article but none of them tell how to use ceph for mail server will be like.
The most close to it: https://www.7layer.org/?p=313, but it was glusterFS
So is there any different when ...
4
votes
1answer
2k views
High availability for ceph storage?
Do ceph have high availability, I config 2 node like this
cluster:
id: 07df97db-f315-4c78-9d2a-ab85007a1856
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
...
0
votes
1answer
42 views
How to benchmark small file (5kb) in ceph
I have read this article: https://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance
But the problems is they benchmark a 4MB file while i need like to benchmark 1000 5kb file
Is ...
1
vote
2answers
660 views
GlusterFS or Ceph RBD for storing Virtual Machine image
I am using glusterfs 5.3 for storing images of virtual machines in Cloudstack/KVM environment, majority of VMs are DB Servers (Sql Server & MariaDB). But I am facing performance issue on VMs ...
2
votes
0answers
466 views
Unable to resize Ceph RBD PVC in Kubernetes
So I have a 4 node (VMs) Kubernetes cluster spun up with Kubespray. I have a Ceph cluster set up from Proxmox, and a pool is available to k8s. I can make deployments using Ceph just fine.
But when ...
0
votes
1answer
678 views
Ceph installation experiences high swap usage
Currently I'm running a 8 server Ceph setup consisting out off 3 Ceph monitors and 5 Ceph nodes. Performance wise the cluster runs great but after time the nodes start swapping the ceph-osd process to ...
2
votes
1answer
688 views
Proxmox Ceph OSD Partition Created With Only 10GB
How do you define the Ceph OSD Disk Partition Size?
It always creates with only 10 GB usable space.
Disk size = 3.9 TB
Partition size = 3.7 TB
Using ceph-disk prepare and ceph-disk activate (See ...
3
votes
1answer
322 views
What is wrong with my initial setup to install ceph?
I have downloaded ceph-ansible to install ceph for an evaluation and I ran on an error with package dependencies and I can't find anything about how to fix this using either ceph-ansible or yum.
The ...
0
votes
1answer
590 views
Compaction error: Corruption: block checksum mismatch: expected 862584094, got 1969278739
Currently trying to handle this right on my 3-day memorial weekend :D
Ceph 13.2.4 (Filestore)
Rook 0.9
Kubernetes 1.14.1
https://gist.github.com/sfxworks/ce77473a93b96570af319120e74535ec
My setup ...
2
votes
0answers
617 views
ceph shows wrong USED space in a single replicated pool
We are using ceph version 14.2.0. We have 4 hosts with 24 BlueStore OSDs, each is 1.8TB (2TB spinning disk).
We have only a single pool with size 2 and I am absolutely sure that we are using more ...
0
votes
1answer
489 views
CEPH for video streaming? [closed]
So i have a video streaming app (HLS) and was wondering if using CEPH or GlusterFS would be a good idea vs using plain SAN storage?
(from performance prospective)
In general is software defined the ...
0
votes
1answer
165 views
Indicating multiple clusters with ceph-ansible
I am working through a Ceph course right now (ceph learning path from packtpub). The course is ok, but has a lot of errors, and isn't always the most accurate. The course expects that you use ceph-...
0
votes
1answer
850 views
Is more memory than the required beneficial for Ceph BlueStore OSDs?
I have a cluster of servers, each of them having 128GB or RAM and 6 x 2TB spinning disks dedicated for BlueStore OSDs.
The servers also act like KVM hosts, so they are not dedicated to Ceph.
In the ...
