Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am in the process of upgrading storage for our MogileFS cluster and am using the rebalance and device drain features to migrate data from one set of devices to another. We have about 55 TB on one set of devices that I would like to migrate to a new set of devices 88 TB free.

I have the following policy setup:

[ashinn@mogile2 ~]$ sudo mogadm rebalance settings
             rebal_policy = from_devices=2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028 to_hosts=5,6,7 leave_in_drain_mode=1

But it appear to only drain/rebalance one device at a time:

[ashinn@mogile2 ~]$ sudo mogadm rebalance status
Rebalance is running
Rebalance status:
             bytes_queued = 250755303323
           completed_devs = 
              fids_queued = 7785000
                    limit = 0
             sdev_current = 2005
             sdev_lastfid = 1444986524
               sdev_limit = none
              source_devs = 2016,2028,2007,2013,2012,2022,2008,2001,2024,2017,2023,2025,2009,2015,2006,2026,2021,2020,2019,2010,2027,2004,2018,2014,2002,2011,2003
            time_finished = 0
             time_started = 1340960590
             time_stopped = 0

At this rate, it would take 4 months to drain all the old devices and rebalance to the new ones! Here is a list of the devices I am trying to drain and the new ones added. dev2001 to dev2028 are set to drain and rebalance to all 3 hosts (including the new devices dev2029 to dev2036 on host id 6):

[ashinn@mogile2 ~]$ sudo mogadm device list | grep dev20
dev2001: drain      2018.942 731.216 2750.158
dev2002: drain      2022.452 727.706 2750.158
dev2003: drain      2022.311 727.848 2750.158
dev2004: drain      2022.211 727.947 2750.158
dev2005: drain      1472.550 1277.608 2750.158
dev2006: drain      2022.135 728.023 2750.158
dev2007: drain      2022.139 728.020 2750.158
dev2008: drain      2022.246 727.912 2750.158
dev2009: drain      2022.369 727.789 2750.158
dev2010: drain      2022.191 727.967 2750.158
dev2011: drain      2022.694 727.464 2750.158
dev2012: drain      2022.256 727.902 2750.158
dev2013: drain      2022.117 728.041 2750.158
dev2014: drain      2022.271 727.887 2750.158
dev2015: drain      2021.590 728.568 2750.158
dev2016: drain      2021.499 728.659 2750.158
dev2017: drain      2021.712 728.446 2750.158
dev2018: drain      2021.191 728.967 2750.158
dev2019: drain      2020.846 729.312 2750.158
dev2020: drain      2021.758 728.400 2750.158
dev2021: drain      2021.490 728.668 2750.158
dev2022: drain      2021.217 728.941 2750.158
dev2023: drain      2020.922 729.236 2750.158
dev2024: drain      2019.909 730.249 2750.158
dev2025: drain      2020.503 729.655 2750.158
dev2026: drain      2020.807 729.352 2750.158
dev2027: drain      2021.056 729.103 2750.158
dev2028: drain      2020.487 729.671 2750.158
dev2029: alive      182.120 10818.996 11001.116
dev2030: alive      184.549 10816.567 11001.116
dev2031: alive      185.268 10815.849 11001.116
dev2032: alive      182.004 10819.112 11001.116
dev2033: alive      189.295 10811.821 11001.116
dev2034: alive      183.199 10817.917 11001.116
dev2035: alive      178.625 10822.491 11001.116
dev2036: alive      180.549 10820.567 11001.116

We have already tried tuning queue_rate_for_rebal, queue_size_for_rebal, and the replicate workers.

We did this once using zones across two data centers and the replication was MUCH faster. We were hoping that a rebalance would work much like replication. But at this rate, it seems like marking the old devices as dead to for replication of fids would be faster.

Are there any other ways to speed up a rebalance (such as multiple devices at once) without having to mark devices as dead?

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.