Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
202
questions
1
vote
1
answer
4k
views
Poor performance with rook, ceph and RBD
I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD.
The OSDs are using the same disk as the VM Operating System.
The VM disks are remote (...
0
votes
1
answer
11k
views
Ceph hardware sizing calculator
I would like to calculate the Hardware sizing for a ceph cluster. There are so few references to the sizing, that I try to get these details here in the community.
E.g. what shall i have depending on
...
- 277
1
vote
1
answer
378
views
Ceph with mainstream (desktop) SSDs?
I am searching for an answer about using SSDs drivers, which are build for the mainstream.
Afaik ceph uses for its marketing to buy cheap disks for entprise ready storages. But I cannot find any ...
- 277
1
vote
0
answers
162
views
Ceph Clock Skew
I want to know where I can configure the limit of skew time for my ceph monitors. Also,how does CEPH throw this clock skew error? Specifically from which file and where can I find the file so that I ...
1
vote
0
answers
135
views
CEPH netfilter connection tracking issues
We have a CEPH cluster (Ubuntu 18.04, Luminous) for Openstack images and volumes. As I was taking it into production I found many performance issues, slow OSDs, and throughput down to a trickle; this ...
- 11
2
votes
2
answers
14k
views
MountVolume.MountDevice failed operation with the given Volume ID already exists
Environment:
Kubernetes cluster with 1 master and 3 nodes Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64) (VMWARE VMs)
screenshot from dashboard
Pod (simple nginx image) cannot be mounted ...
- 23
3
votes
1
answer
476
views
Setup ceph for storage mailserver?
I have read some article but none of them tell how to use ceph for mail server will be like.
The most close to it: https://www.7layer.org/?p=313, but it was glusterFS
So is there any different when ...
- 301
4
votes
1
answer
3k
views
High availability for ceph storage?
Do ceph have high availability, I config 2 node like this
cluster:
id: 07df97db-f315-4c78-9d2a-ab85007a1856
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
...
- 301
0
votes
1
answer
188
views
How to benchmark small file (5kb) in ceph
I have read this article: https://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance
But the problems is they benchmark a 4MB file while i need like to benchmark 1000 5kb file
Is ...
- 301
1
vote
2
answers
1k
views
GlusterFS or Ceph RBD for storing Virtual Machine image
I am using glusterfs 5.3 for storing images of virtual machines in Cloudstack/KVM environment, majority of VMs are DB Servers (Sql Server & MariaDB). But I am facing performance issue on VMs ...
- 11
2
votes
0
answers
670
views
Unable to resize Ceph RBD PVC in Kubernetes
So I have a 4 node (VMs) Kubernetes cluster spun up with Kubespray. I have a Ceph cluster set up from Proxmox, and a pool is available to k8s. I can make deployments using Ceph just fine.
But when ...
- 613
0
votes
1
answer
2k
views
Ceph installation experiences high swap usage
Currently I'm running a 8 server Ceph setup consisting out off 3 Ceph monitors and 5 Ceph nodes. Performance wise the cluster runs great but after time the nodes start swapping the ceph-osd process to ...
- 173
2
votes
1
answer
1k
views
Proxmox Ceph OSD Partition Created With Only 10GB
How do you define the Ceph OSD Disk Partition Size?
It always creates with only 10 GB usable space.
Disk size = 3.9 TB
Partition size = 3.7 TB
Using ceph-disk prepare and ceph-disk activate (See ...
- 163
3
votes
1
answer
441
views
What is wrong with my initial setup to install ceph?
I have downloaded ceph-ansible to install ceph for an evaluation and I ran on an error with package dependencies and I can't find anything about how to fix this using either ceph-ansible or yum.
The ...
- 151
0
votes
1
answer
2k
views
Compaction error: Corruption: block checksum mismatch: expected 862584094, got 1969278739
Currently trying to handle this right on my 3-day memorial weekend :D
Ceph 13.2.4 (Filestore)
Rook 0.9
Kubernetes 1.14.1
https://gist.github.com/sfxworks/ce77473a93b96570af319120e74535ec
My setup ...
- 157
2
votes
0
answers
1k
views
ceph shows wrong USED space in a single replicated pool
We are using ceph version 14.2.0. We have 4 hosts with 24 BlueStore OSDs, each is 1.8TB (2TB spinning disk).
We have only a single pool with size 2 and I am absolutely sure that we are using more ...
- 131
0
votes
1
answer
859
views
CEPH for video streaming? [closed]
So i have a video streaming app (HLS) and was wondering if using CEPH or GlusterFS would be a good idea vs using plain SAN storage?
(from performance prospective)
In general is software defined the ...
- 109
0
votes
1
answer
238
views
Indicating multiple clusters with ceph-ansible
I am working through a Ceph course right now (ceph learning path from packtpub). The course is ok, but has a lot of errors, and isn't always the most accurate. The course expects that you use ceph-...
- 2,747
0
votes
1
answer
1k
views
Is more memory than the required beneficial for Ceph BlueStore OSDs?
I have a cluster of servers, each of them having 128GB or RAM and 6 x 2TB spinning disks dedicated for BlueStore OSDs.
The servers also act like KVM hosts, so they are not dedicated to Ceph.
In the ...
- 131
3
votes
1
answer
612
views
Libvirt pool cannot see or create rbd clones
While trying to get a cloned disk running from my OS snapshot I run into the problem that Libvirt cannot see existing images cloned from a snapshot. Created via:
$ rbd -p vmdisks clone vmdisks/...
- 141
1
vote
0
answers
271
views
ceph-deploy osd create --data /dev/vdb fails
From my ceph (mimic release) administrative node I ran
ceph-deploy osd create --data /dev/vdb ceph0
against a bare metal ceph node and it worked without error.
Now I run the same command against a ...
- 942
1
vote
0
answers
569
views
Hung kernel tasks after unclean shutdown of ceph cluster
I am running ceph (created by the rook-ceph operator v0.9.3) on kubernetes v1.13. After an unclean shutdown of our cluster, some processes randomly go into uninterruptible sleep. After some time, the ...
- 11
0
votes
1
answer
932
views
Ceph - Erasure Coded Pools - Always get inactive pgs
I'm trying to achieve similar thing to raid6 on ceph.
But When I'm creating erasure coded pools (k=3 + m=2(or k=4) ) I always get inactive pgs.
ceph health details are:
HEALTH_WARN Reduced data ...
- 309
1
vote
1
answer
2k
views
Ceph iSCSI Gateway installation on Ubuntu 18.04
I'm installing Ceph using Ansible with ceph-ansible project, branch static-3.2. There is a problem with iSCSI Gateway installation.
If you are using iscsigws name in inventory file - it shows that ...
- 309
0
votes
0
answers
29
views
How do I use a non-standard ceph system user?
I configured a ceph user on my cluster named "cepher." I ran ceph-deploy as this user to deploy some servers.
Then I see this:
[block][INFO ] Running command: sudo ceph --cluster ceph --name client....
- 942
1
vote
0
answers
72
views
How wide (data + parity) would you go for erasure coded pools / raid 6?
I am currently doing k=8 and m=2
or in other words data = 8 + parity = 2 shards
I am using an all hdd ceph cluster.
I have thought about going data = 11 + parity = 2 which would save a lot of ...
- 176
0
votes
1
answer
467
views
Ceph Erasure Profile with k+m> 6 disks -> PGs to be stuck on creating+incomplete state forever
So my pools has 5 nodes with 12 osds each of 8tb.
Currently I am trying to create an erasure coded pool of k=8 m=2, however after adding this profile and create an ecpool with this profile the pools ...
- 176
1
vote
2
answers
7k
views
How to add Hetzner Cloud Disk Volume to extend /var partition?
I have a new Hetzner cloud server and added an extra 50GB disk volume (CEPH) to my server. The default disk is only 40GB. I thought it would be added to the default storage like the traditional ...
- 139
3
votes
1
answer
4k
views
Ceph or Gluster for implementing big NAS
We are planning to build NAS solution which will be primarily used via NFS and CIFS and workloads ranging from various archival application to more “real-time processing”. The NAS will not be used as ...
- 33
0
votes
1
answer
131
views
mount cephfs or rbd via Internet
Is there any suitable solution for this?
I know about security issues, but is it possible?
Or other better storage solutions?
We tested s3fs+cephrgw, but It's very slow and unstable.
- 1
1
vote
1
answer
552
views
Ceph architecture for small HPC cluster
I want to build Ceph Storage Cluster for HPC use. (CentOS 7 based)
For now I have enterprise SAS RAID enclosure with 3 shelves by 12 4TB disks(36 total).
Now it is configured as default RAID6 rig. And ...
- 163
0
votes
1
answer
370
views
How to upgrade librbd1 and librados2 for oVirt 4.2.x (node ng)
oVirt 4.2 comes with librbd1 und librados2 from the Ceph Hammer release which is 0.94.5.
I need to update both libraries to the luminous version which is 12.x. because my ceph server is not able to ...
- 478
0
votes
1
answer
844
views
Why does ovirt VM using ceph disk stay in "waiting for launch" status
My setup comprises of ceph mimic (centos 7, setup with ceph-ansible), cinder/keystone combo on pike release, and ovirt 4.2.5.1.
The external cinder provider is setup and I can create disks.
When ...
- 478
2
votes
1
answer
330
views
Ceph status health ok, but has flag "nearfull"
Recently in Ceph there was a status of WARN because 3 disks were 85-87% full. I expanded the cluster by adding the server to the storage. But now I see flag "nearfull". Previously, this wasn't see.
...
- 323
1
vote
0
answers
48
views
openstack instance's image name lost after integration with ceph rbd
after integrated with ceph rbd,my openstack could not show "image name" of my instance, is there any option that controlling this feature?
- 189
1
vote
0
answers
118
views
Tried to recover a Ext4 Superblock to a CEPH disk
We have a server with Proxmox installed with Debian, it's an dedicated server (sda). The VM's are being backed up every night on a CEPH disk (sdb). After an reboot of the dedicated, the ext4 drive ...
- 11
17
votes
1
answer
13k
views
What is the difference between Ceph and NFS when it comes to shared file storage?
Can someone explain to me what is Ceph compared to NFS?
From a shared file storage perspective? What is the added value of using Ceph instead of NFS?
- 2,140
1
vote
0
answers
584
views
Slow samba write speed to ceph backend
I have backend on cephfs cluster, mounted on gateway. Mount path is shared with samba 4.7.4. But my speed is low, about 1,5MB/s with small files (1k - 100k). I copy ot with win server 2008 client.
...
- 11
1
vote
2
answers
2k
views
RAID-0 stripe size for Ceph OSD
I have 3 servers that I will use for a new Ceph cluster. It's my first Ceph "playground"... Each server has 2x1TB and 6x2TB HDDs connected to two separate 4-channel SAS controllers, each with 1GB ...
- 131
1
vote
0
answers
769
views
Ceph is full but disk is almost empty
I have a ceph cluster with 2 nodes and 3 osd's each. Each osd is on one partion on a 8TB disk. The server is limited in amount of disk so I even have system on the same disks.
On install I used ...
- 11
2
votes
1
answer
1k
views
On what network do the clients connect to the CEPH cluster (public/private)?
It's recommended to have a public network and cluster network when setting up CEPH.
For what I understand this cluster network is what the nodes use to replicate data accross so that would preferably ...
- 239
1
vote
1
answer
311
views
CephFS needed if you only need block storage?
I'm looking into building a CEPH cluster as a storage solution for our proxmox cluster.
It's unclear to me if I need to set up a metadata server too, because this is needed for CephFS but I think ...
- 239
9
votes
3
answers
8k
views
Is Ceph possible to handle hardware RAID arrays (LUNs) as OSD drives?
I am pretty new to Ceph and try to find out if Ceph supports hardware level raid HBAs.
Sadly could not find any information. What I found is, that it is recommended to use plain disks for OSD. But ...
- 277
3
votes
1
answer
1k
views
CTDB Samba failover not highly available
TL;DR
Failing a node in a ctdb + samba cluster while interacting with a share interrupts share connection.
here and here state there is work in progress to make this possible
here states it is ...
0
votes
2
answers
3k
views
install grub in a usb and boot proxmox from another drive
Good morning my friends!
I'm tryng a solution before buy some new hardware.
So this is my situation:
I have some beautifull hp dl360p gen8 that comes with an p420i hardware raid controller.
What ...
- 111
3
votes
1
answer
220
views
Which distributed filesystem will actually fit my needs?
Hear me out
I have seen the question asked (in different forms) here, here, and perhaps the best one I found was here, but I do not think this is a duplicate because quite some time has past since ...
- 143
2
votes
1
answer
176
views
Virtual Hosting Cluster File System Confusion
My title probably doesn't encompass the full scope of what I'm needing, so I'll lay out what I want to accomplish.
I have two Linux servers with large drive arrays, multiple CPUs and a large amount ...
- 107
1
vote
1
answer
824
views
Proxmox Ceph OSD Fault
I am running Proxmox with 2 nodes and Ceph. Recently I had a drive die and replaced it. In the process of trying to initialize it as an OSD, all I get are fault errors:
2017-07-31 15:05:05.278499 ...
- 21
7
votes
1
answer
4k
views
Better performance when HDD write cache is disabled? (HGST Ultrastar 7K6000 and Media Cache behavior)
Attention, please. Long read.
During initial performance tests of Hitachi Ultrastar 7K6000 drives that I'm planning to use in my Ceph setup I've noticed a strange thing: write performance is better ...
- 91
1
vote
2
answers
405
views
broken ceph dependency with centos7 and epel
Okay, I'm trying to install ceph, using just "yum install ceph" from the epel repositories.
This is what I get:
--> Finished Dependency Resolution
Error: Package: 1:python-cephfs-0.80.7-0.8.el7....
- 1,406