Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [ceph]

Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.

18
votes
1answer
3k views

Cause of page fragmentation on “large” server with xfs, 20 disks and Ceph

Any insight from someone with a bit of experience in the linux IO system would be helpful. Here is my story: Recently brought up a cluster of six Dell PowerEdge rx720xds to serve files via Ceph. ...
8
votes
2answers
4k views

CEPH's raw space usage

I can't understand ceph raw space usage. I have 14 HDD (14 OSD's) on 7 servers , 3TB each HDD ~ 42 TB raw space in total. ceph -s osdmap e4055: 14 osds: 14 up, 14 in pgmap v8073416: ...
7
votes
3answers
1k views

is ceph possible to handle hardware RAID arrays (LUNs) as OSD drives?

I am pretty new to ceph and try to find out if ceph supports hardware level raid HBAs. Sadly could not find any information. What I found is, that it is recommended to use plain disks for OSD. But ...
6
votes
1answer
1k views

What is the difference between Ceph and NFS when it comes to shared file storage?

Can someone explain to me what is Ceph compared to NFS? From a shared file storage perspective? What is the added value of using Ceph instead of NFS?
4
votes
1answer
575 views

How can I measure complex I/O activity to develop realistic benchmarking? [closed]

I need to retrieve a few metric from my service server to benchmark another stroage solution. There are various options are provied by benchmark program(I choosed FIO,http://freecode.com/projects/fio)....
4
votes
3answers
2k views

Does it work to run mysql with the data directory on a CephFS mount?

We have a database cluster that uses MySQL on DRBD with Pacemaker+Corosync, which is great. My question is whether it is possible (and whether anyone has been successful) to mount a CephFS drive to /...
4
votes
1answer
2k views

Which iscsi target on linux for serving vSphere? [closed]

All such questions are a bit dated; I'm hoping things have changed in the past year. What do you guys use to serve a DRS cluster with vSphere 5.1/5.5? SCSI-3 persistent reservation support would be ...
4
votes
0answers
1k views

Better performance when HDD write cache is disabled? (HGST Ultrastar 7K6000 and Media Cache behavior)

Attention, please. Long read. During initial performance tests of Hitachi Ultrastar 7K6000 drives that I'm planning to use in my Ceph setup I've noticed a strange thing: write performance is better ...
3
votes
2answers
136 views

Should io and cpu intensive servers be separated in kubernetes cluster?

We are designing a new cluster architecture for our web service and are planing to use Ceph object storage and kubernetes for our services. for optimizing our servers We have different options: Use ...
3
votes
1answer
3k views

ceph - can't start osd on rebooted cluster host

I've rebooted the server (one of ceph's hosts). Started the cluster, but the osd, that's on the host, which was rebooted is down. The osd's # is 2, so when I try: sudo /etc/init.d/ceph start osd.2 ...
3
votes
1answer
5k views

Ceph OSD always 'down' in Ubuntu 14.04.1

I am trying to install and deploy a ceph cluster. As I don't have enough physical servers, I create 4 VMs on my OpenStack using official Ubuntu 14.04 image. I want to deploy a cluster with 1 monitor ...
3
votes
1answer
3k views

Why is GlusterFS so slow here?

We've set up a mirroring pair of GlusterFS servers. No special tuning, whatever came "out of the box" with GlusterFS-3.5.1 in the official RHEL6 RPM, that's what we have. The cluster works, but the ...
3
votes
1answer
45 views

Shared and replication filesystem support POSIX

I'm looking for opensource solution that support my use case. I have now 4 node on my cluster network and i need this. Store file system (huge list) Replication my file save on one node and replicate ...
3
votes
1answer
519 views

CTDB Samba failover not highly available

TL;DR Failing a node in a ctdb + samba cluster while interacting with a share interrupts share connection. here and here state there is work in progress to make this possible here states it is ...
3
votes
0answers
855 views

ceph osd down and rgw Initialization timeout, failed to initialize after reboot

Centos7.2, Ceph with 3 OSD, 1 MON running on a same node. radosgw and all the daemons are running on the same node, and everything was working fine. After reboot the server, all osd could not ...
2
votes
1answer
99 views

Virtual Hosting Cluster File System Confusion

My title probably doesn't encompass the full scope of what I'm needing, so I'll lay out what I want to accomplish. I have two Linux servers with large drive arrays, multiple CPUs and a large amount ...
2
votes
3answers
1k views

Scale-out distributed storage with snapshots

I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster ...
2
votes
1answer
340 views

What kind of “volume/storage management” has the largest support/feature set these days? [closed]

(DISCLAIMER: This is "time bound" as probably every other question, I just point it out now instead of implicating it) part of reviewing a few basic decisions for our infrastructure is once again the ...
2
votes
1answer
115 views

Ceph status health ok, but has flag “nearfull”

Recently in Ceph there was a status of WARN because 3 disks were 85-87% full. I expanded the cluster by adding the server to the storage. But now I see flag "nearfull". Previously, this wasn't see. ...
2
votes
1answer
2k views

DELL FS12-SC PERC 6/i replacement with JBOD support? [closed]

I have 3 DELL FS12-SC servers with a bunch of 3TB drives (issue number 1, the perc 6/i doesn't appear to support 3TB drives). I'm setting up a ceph cluster and need to run the drives in JBOD mode. The ...
2
votes
1answer
4k views

copy kvm image from ceph to other storage

A while ago I created a ceph storage cluster and connected it to proxmox2 according to the method described in http://pve.proxmox.com/wiki/Storage:_Ceph. We have some kvm images running on the ceph ...
2
votes
1answer
108 views

Which distributed filesystem will actually fit my needs?

Hear me out I have seen the question asked (in different forms) here, here, and perhaps the best one I found was here, but I do not think this is a duplicate because quite some time has past since ...
2
votes
0answers
295 views

What is the correct usage of ceph-objectsore-tool?

I am trying to export data for pg, but am getting errors from ceph-objectstore-tool. I see some examples online of usage which seem to match this, and reading through the usage statement that the ...
2
votes
0answers
542 views

Slow fsync() with ceph (cephfs)

I have built an experimental ceph cluster - 12 nodes, 50 osds, 3 mons, 3 mds, for which I'm trying to run a samba gateway. It seems that when writing lots of small files, the fsync() system calls from ...
2
votes
1answer
181 views

Make ceph minimize spread of file parts over OSDs

I am considering an option of ceph as distributed filesystem for my home-made MAID (massive array of idle drives). As far as I understand, Ceph oriented for cluster use and spread data evenly over ...
2
votes
0answers
1k views

Conditions for “mds cluster is degraded”

I see into Ceph and after running command $ ceph status I meet text "mds cluster is degraded". I try to seek it at official Ceph docs, but nothing find. If I understand correctly, MDS needed for ...
1
vote
4answers
14k views

Unable to add initial monitor to Ceph in Ubuntu

I am trying to set up a Ceph cluster. I have 4 nodes - 1 admin-node, 1 monitor and 2 object storage devices. The installation guide I am using is given at the following location: http://ceph.com/docs/...
1
vote
1answer
6k views

How do I mount an rbd device from fstab

According to this link: http://docs.ceph.com/docs/master/start/quick-rbd/ I can mount an rbd, which works perfectly. The question I have is how do I do this from fstab? The end goal being to mount ...
1
vote
1answer
583 views

Use Ceph as redundant storage for failover FTP service?

I need to setup a redundant FTP storage service that can survive one server-crash. I have two servers SrvA and SrvB that shares a virtual IP IpVirt (an IP that will usually point to SrvA but that ...
1
vote
2answers
2k views

Is ceph replication based on nodes or disks?

Im currently evaluating storage systems for xenserver. Because data replication is important in the case of a failure i have a question regarding replication in ceph. As far as i know every disk in a ...
1
vote
1answer
3k views

Will running Ceph (or similar systems) virtualized degrade its performance?

I am setting up a Ceph cluster. The client is asking for it to be done in virtual machines, one hypervisor / VM per server. Given my previous (minor) experience with virtual machines, I wonder if ...
1
vote
1answer
2k views

How do I mount one of multiple filesystems in a ceph cluster?

Ceph now includes (experimental) support for multiple filesystems within a single storage cluster; but the mount options don't seem to allow specifying which filesystem to mount. I have configured ...
1
vote
2answers
2k views

How do I mount rbd devices with the fuse driver?

This is a follow-up question from: How do I mount an rbd device from fstab Which I asked some time back. How do I mount rbd devices with the fuse driver? (I can't use the kernel driver from docker ...
1
vote
2answers
193 views

broken ceph dependency with centos7 and epel

Okay, I'm trying to install ceph, using just "yum install ceph" from the epel repositories. This is what I get: --> Finished Dependency Resolution Error: Package: 1:python-cephfs-0.80.7-0.8.el7....
1
vote
2answers
3k views

Is Ceph too slow and how to optimize it?

The setup is 3 clustered Proxmox for computations, 3 clustered Ceph storage nodes, ceph01 8*150GB ssds (1 used for OS, 7 for storage) ceph02 8*150GB ssds (1 used for OS, 7 for storage) ceph03 8*250GB ...
1
vote
1answer
453 views

On what network do the clients connect to the CEPH cluster (public/private)?

It's recommended to have a public network and cluster network when setting up CEPH. For what I understand this cluster network is what the nodes use to replicate data accross so that would preferably ...
1
vote
1answer
173 views

CephFS needed if you only need block storage?

I'm looking into building a CEPH cluster as a storage solution for our proxmox cluster. It's unclear to me if I need to set up a metadata server too, because this is needed for CephFS but I think ...
1
vote
1answer
504 views

Proxmox Ceph OSD Fault

I am running Proxmox with 2 nodes and Ceph. Recently I had a drive die and replaced it. In the process of trying to initialize it as an OSD, all I get are fault errors: 2017-07-31 15:05:05.278499 ...
1
vote
1answer
149 views

Can VM hard disk size be bigger than single SSD in Ceph Storage?

I have a 3 node ceph proxmox cluster for storage with node1 7*143GB ssds, node2 7*143 ssds, node3 7*218 ssds. Now I have read through the http://docs.ceph.com/docs/jewel/architecture/#mapping-pgs-to-...
1
vote
1answer
2k views

Ceph Pool Snapshot for CephFS Backup and/or Data Recovery

Since CephFS does not support snapshot yet, could we use Ceph Pool Snapshot for backup purpose from accidental deletion of files inside CephFS? ceph osd pool mksnap {pool-name} {snap-name} ceph osd ...
1
vote
1answer
714 views

After initial deployment Ceph cluster stays in active+degraded state

I've created a small Ceph cluster following quick start guide with one exception, I've used a separate disk for OSDs rather than a folder. Instead of ceph-deploy osd prepare node2:/var/local/osd0 ...
1
vote
0answers
49 views

Hung kernel tasks after unclean shutdown of ceph cluster

I am running ceph (created by the rook-ceph operator v0.9.3) on kubernetes v1.13. After an unclean shutdown of our cluster, some processes randomly go into uninterruptible sleep. After some time, the ...
1
vote
0answers
20 views

How wide (data + parity) would you go for erasure coded pools / raid 6?

I am currently doing k=8 and m=2 or in other words data = 8 + parity = 2 shards I am using an all hdd ceph cluster. I have thought about going data = 11 + parity = 2 which would save a lot of ...
1
vote
1answer
907 views

Ceph or Gluster for implementing big NAS

We are planning to build NAS solution which will be primarily used via NFS and CIFS and workloads ranging from various archival application to more “real-time processing”. The NAS will not be used as ...
1
vote
1answer
69 views

Ceph architecture for small HPC cluster

I want to build Ceph Storage Cluster for HPC use. (CentOS 7 based) For now I have enterprise SAS RAID enclosure with 3 shelves by 12 4TB disks(36 total). Now it is configured as default RAID6 rig. And ...
1
vote
0answers
30 views

openstack instance's image name lost after integration with ceph rbd

after integrated with ceph rbd,my openstack could not show "image name" of my instance, is there any option that controlling this feature?
1
vote
0answers
30 views

Tried to recover a Ext4 Superblock to a CEPH disk

We have a server with Proxmox installed with Debian, it's an dedicated server (sda). The VM's are being backed up every night on a CEPH disk (sdb). After an reboot of the dedicated, the ext4 drive ...
1
vote
0answers
252 views

Slow samba write speed to ceph backend

I have backend on cephfs cluster, mounted on gateway. Mount path is shared with samba 4.7.4. But my speed is low, about 1,5MB/s with small files (1k - 100k). I copy ot with win server 2008 client. ...
1
vote
0answers
287 views

Ceph is full but disk is almost empty

I have a ceph cluster with 2 nodes and 3 osd's each. Each osd is on one partion on a 8TB disk. The server is limited in amount of disk so I even have system on the same disks. On install I used ...
1
vote
0answers
145 views

Using NFS Mounts as Long Term Container Storage

this is more of a question of best practice than anything. I currently have a proxmox clustered deployment of three servers, all accessing a ceph cluster (that is self hosted on the same servers). ...