Questions tagged [disaster-recovery]
Disaster recovery and preparedness is an unfortunate aspect of systems administration. This tag should be used for help with planning, implementation and best-practices related to recovering from a catastrophic event on a server or in a datacenter environment.
358
questions
1
vote
3
answers
61
views
How to setup recovery on Ubuntu server?
We have Ubuntu server on the remote location that may or may not have an Internet access. It will run several applications and we want to make sure that if something goes wrong on OS or app level, we ...
- 111
0
votes
1
answer
83
views
How to reroute from one subdomain to another in AWS Route53?
I have two hosted zones (main.mydomain.com and backup.mydomain.com). They include similar records (more than 100 records). Their records only differ in value/route traffic to. The main subdomain ...
- 103
0
votes
0
answers
12
views
ADO on premise recovery scenario (domain controller, app tier + db tier)
We are thinking about disaster recovery scenarios of one of our services and I am currently lacking some knowledge or sources to further investigate.
We have an Azure DevOps On Premise server that has ...
- 103
0
votes
1
answer
326
views
Multi-Master K8S cluster fails when half of the masters are down
i have a 4 masters HA K8S cluster , (accross 2 datacenters , 2 in each site) , but the kubectl command stops working after shuting down 2 masters , is this the expected behaviour?
I want the cluster ...
- 3
1
vote
2
answers
287
views
GCP disk snapshots: Would losing the full snapshot render the incremental snapshots useless?
On GCP, when you do periodic snapshots of disks, the first snapshot is a full snapshot of the disk and the following snapshots consists of only the deltas of the previous snapshots. If a snapshot is ...
google-cloud-platform
0
votes
1
answer
152
views
lvm recovery with raid reconfiguration
I have a Dell R510 with a H200 controller previously and I have two disks: a 2T SAS and a 250G SSD. I created two raid0s using those two disks on H200 and then created lvm on top of it. Here is my ...
- 1
2
votes
1
answer
158
views
ESXi Host backup including extra vibs
I have a licensed ESXI Host with a couple of extra vibs (Chelsio, NVidia, dell) installed. I also have vcenter standard. More that I like to I discover that I have to reinstall the host, e.g. VMware ...
- 21
0
votes
1
answer
202
views
Disaster Recovery: PXE boot Windows Server 2019 from ISO
I have a dedicated physical Windows Server 2019 on the Ionos cloud platform. I need to implement a disaster recovery plan. I also have added a shared storage block on another Ionos server and mapped ...
2
votes
3
answers
1k
views
Need to recover RAID 6 array
By definition
RAID 6 is an array of independent hard drives equipped with two
independent and distributed checksum schemes
Does that mean I can recover each of the hard drives one by one using ...
- 121
0
votes
0
answers
145
views
Rescue Tip: Wrong library file installation damaged Ubuntu 18.04 after upgrade
There was an old Ubuntu 14.04 linode.com instance which I attempted to upgrade to 18.04. Some Dependencies where somehow not upgraded anyway.
So I attempted to install corresponding deb file from ...
- 33
8
votes
1
answer
2k
views
Does one failed drive + one single bad sector destroy an entire RAID 5?
During planning my RAID setup on a Synology Disk Station I've done a lot of reading about various RAID types, being this a great reading: RAID levels and the importance of URE (Unrecoverable Read ...
- 275
0
votes
1
answer
77
views
If I image a virtual machine with SQL server installed will the database work on a new instance?
On AWS I've always imaged machines that contained SQL Server installations and running databases. When I spin up new instances from the images, SQL Server and all database always returned without ...
- 1
0
votes
1
answer
76
views
desperate attempt to head off RAID 0 failure, can i copy a disk with dd?
A colleague has been using a 10TB RAID 0 array to edit a film. He suddenly realized it's 98% full. The array has started seriously acting up, but cooperates grudgingly if you shut down all processes ...
- 516
2
votes
1
answer
163
views
Systemd and Disaster-Recovery Stand-By systems
We're using systemd to run various services in production. (Duh...)
We're building out a matching "disaster-recovery" site, which will have the same application installed -- with the same ...
- 2,282
0
votes
2
answers
42
views
Cloud definition met? [closed]
note: question has been reviewed after suggestions it could be opinion based
We recently suffered an outage due to a popular big provider's datacenter fire.
Our public cloud instances hosted in that ...
- 1,699
0
votes
1
answer
32
views
Disaster recovery process document for Google cloud Virtual Machine
Hello we have been directed by auditors to provide disaster recovery document for Google Cloud Virtual Machine. Please Is this provided by Google? If so how can I get it. I would appreciate resources ...
1
vote
1
answer
584
views
Safe to delete managed incremental snapshots on Azure?
Incremental snapshots of managed disks on Azure is a relatively new thing, and is not completely documented, so I have a crucial question:
If I have incremental snapshots A, B, C and D of a disk (the ...
0
votes
0
answers
29
views
How to make backup for the Local Server
I have critical server that running the local company system, that have system backup and company files and business documents with critical and important severity.
I'm planning to create a Network ...
1
vote
0
answers
66
views
Setting UP disaster recovery , one region to multi region in Google Cloud
I'm new to GCP and also learning about Multi-Regional architectures .
We already have single region ( but multi zonal ) infrastructure setup.
DNS names of the internal components do not have region/...
0
votes
1
answer
143
views
promox is blocked from fully booting by ifupdown2 start job
i have a promox server with ifupdown2 installed.
recently when trying to reboot the entire server promox failed to get to console, it gets stuck on replication runner failing and the ifupdown2 service ...
- 111
1
vote
0
answers
262
views
Is it possible to backup and restore a Netbackup Master server using Windows Server Backup?
I have installed Netbackup 8.2 master server on Windows Server 2019 system.
I read this document , which says you neet to re-install OS and Netbackup itself, then the catalog in order to restore ...
0
votes
0
answers
213
views
Can't reconnect VPS Essential OVH after removing set_hostname from cloud-init
I have a VPS Essential in OVH with Ubuntu 18.04 LTS installed.
I just removed the set_hostname module of the cloud_init_modules from /etc/cloud/cloud.cfg and reboot the system and now I cannot connect ...
- 113
1
vote
1
answer
179
views
How can I recover data from a five disk raid 5 on HP Proliant DL 380 [closed]
How can I recover data from a five disk RAID 5 on HP Proliant DL 380 where four of the five disks are reporting a SMART error. I have some critical data on the failed logical drive and checking online ...
- 11
0
votes
1
answer
128
views
Azure VM off-site backups via CLI
I'm currently backing up Azure VMs via Azure CLI:
create resource group:
az group create -n backup-resource-group -l uksouth
create recovery services vault:
az backup vault create --resource-group ...
- 183
7
votes
2
answers
2k
views
In 2020 - are there any viable Linux block-level replication alternatives for DRBD? [closed]
I'm researching how can we implement near-realtime replication from primary datacenter to a disaster recovery site. Data that would get replicated would be:
Images of KVM VMs
MySQL and PostgreSQL ...
- 29.9k
1
vote
1
answer
543
views
Windows 2019 backup: bare metal restore
I'm rehearsing a virtual machine bare metal restore.
Backups are taken daily and saved on a secure external server.
We are assuming a total disaster, so bad that no domain controller is available, so, ...
- 422
0
votes
2
answers
751
views
Copy binary file from other system with builtin shell commands only
I accidentally just have removed critical shared library and no one dynamically linked executable can't be run now. For example, even /bin/ls shows error while loading shared libraries: libunwind.so....
- 103
0
votes
1
answer
133
views
AWS DR Setup and runbook
I'm working on creating a DR setup and runbook based on AWS.
I don't have any experience with creating DR setups so it would be really helpful if the experienced veterans can guide me through it.
...
- 391
1
vote
0
answers
252
views
Zerto Failover - Win 2012/16 Servers Get Stuck on "Getting Devices Ready"
We are using Zerto to replicate our VMs to an offsite datacenter. During our failover tests the VMs have a tendency to "hang" from 15 minutes to several hours on "Getting Devices Ready." I have read ...
- 11
-1
votes
1
answer
1k
views
Does Barman need ssh connection setup when installed on the same host as the PostgreSQL host?
I'm trying to configure Barman (pgbarman) to work alongside the PostgreSQL database on the same host, i don't want to install barman on a separated host, but I can't find any documentation about such ...
- 1
6
votes
1
answer
291
views
Recovering lost VHDX / VM (deleted by Veeam)
Earlier this week a VM on one of our hypervisors experienced extended downtime (~24 hours) due to some Windows updates going wrong. I ultimately was able to fix the issue, and noticed yesterday that ...
- 61
0
votes
1
answer
782
views
Data recovery failure from Azure Data Recovery Vault using MARS Agent with on-premises computer
Despite trying several times since last night, the recovery of a volume from an Azure Data Recovery Vault to a new location on the original source computer fails after approximately (but not exactly) ...
- 141
0
votes
0
answers
166
views
moving raid 5 hard disks from ibm x3500 m1 to ibm x3650 m1
one of our server ibm x3500 m1 type 7977 dead yesterday. can we move its hard drive to access data on raid 5 logical drive on ibm x3650 m1 server with same serveRaid 8K raid controller?
Usman
9
votes
3
answers
3k
views
Database accidentally deleted with a bash script [duplicate]
Edit: a follow-up question: Restore mongoDB by --repair and WiredTiger.
My developer committed a huge mistake and we cannot find our Mongo database anywhere in the server.
He logged into the server, ...
- 327
1
vote
2
answers
2k
views
BTRFS unmountable after cold reboot (total_rw_bytes is twice too big)
One of my users in research environment invoked out-of-memory on a server which mounts a 52TB btrfs partition. I had to power cycle the server.
After the reboot my btrfs partition cannot be mounted in ...
- 373
1
vote
1
answer
62
views
Azure VM Site Replication
I'm looking to set up site replication for our Azure VM's. My question is this, how often are the VM's backed up/replicated to the DR site? Is this replication automated or would I need to kick off ...
- 678
0
votes
1
answer
147
views
What happens to data in a GCP project if all owners or the project itself is suspended?
I have realized recently that a personal domain on google apps for business got suspended (the free for life version that was available in beta). This actually happened like 6 months ago and I didn't ...
2
votes
1
answer
2k
views
Recover ZFS data
Context/Hardware:
HP microserver gen8
1x1TB - standalone, 2x4TB Raid
1x16GB iLO SDCARD with Debian + OpenMediaVault
Event:
SDCARD failure
restarted server and installed Ubuntu on 1TB drive
...
- 188
1
vote
1
answer
235
views
btrfs recover: reliable or last-straw?
The chunk tree of a btrfs filesystem corrupted and I could not recreate it after hour-long reconstruction. As a last resort (besides restoring from backup) I could use btrfs restore -S -x -m -v to get ...
- 633
1
vote
0
answers
61
views
Recovering VCS 2 node cluster in DR?
We are looking for a solution to recover the 2 node VCS Cluster at DR location for a DR Test.
Prod: Hardware is Dell with 24 cores 64 GB RAM with Suse 11.4 Linux over SAN Boot & 4 NIC's of 1 Gig.
...
- 87
0
votes
1
answer
55
views
Can I shadowcopy/backup the AD search index?
We have a 1TB+ search index in our Server 2012 Active Directory (6.1) domain. This is located on a NAS, so rebuilding it takes "forever".
For DR purposes, is there a way to correctly back up the ...
- 146
1
vote
0
answers
107
views
Tried to recover a Ext4 Superblock to a CEPH disk
We have a server with Proxmox installed with Debian, it's an dedicated server (sda). The VM's are being backed up every night on a CEPH disk (sdb). After an reboot of the dedicated, the ext4 drive ...
- 11
0
votes
0
answers
35
views
Possibility to recover from majority of C:\windows being deleted
So I have an interesting and somewhat fun scenario. A developer was working on some scripts that have worked in the past when run by him which deleted files in the working directory, but supposedly ...
1
vote
1
answer
278
views
Can I restore a SQL Server instance from file backups only?
Our windows 2016 server failed to restart after a windows update yesterday.
In order to avoid the update that crashed it issue (whatever it was), I'm in the process of rebuilding it clean.
So I've ...
- 111
0
votes
2
answers
836
views
Recover a failing node on Percona 5.5.41-37.0-55
I have a 3 node percona cluster of where 1 of the nodes has, because of a server crash, InnoDB log sequence problems.
180425 17:32:53 InnoDB: Error: page 151555 log sequence number
12431827692
...
- 3
0
votes
0
answers
72
views
Recovering software raid after a hardware raid array was created on the disks
I had a server with 4 x 500GB SSDs in a software RAID10 array. I moved these drives to a new machine that had an HP P410i Hardware RAID controller. I didn't want to use the hardware raid controller, ...
- 11
0
votes
1
answer
75
views
Azure Site recovery
Planning for ASR and migrating all our servers to Azure in distant future from VMware. Issue is that we do not have any VPN capable device in the network and have read that it is necessary.
Can ...
- 1
1
vote
2
answers
529
views
What would cause mysqlcheck to incorrectly report a table as undamaged?
We are administering a MySQL server for one of our customers that has >100 databases with about 50 tables each, many of them InnoDB tables. The server crashed and I'm trying to find the culprit. When ...
- 213
0
votes
0
answers
3k
views
Synology: How to restore data from an accidentally deleted volumeX (BTRFS)?
The Synology in question has a RAID6 configured via mdadm which features a BTRFS filesystem mounted to /volume1. This path was mounted into a Debian chroot. The unmount was configured in the chroot ...
- 191
2
votes
1
answer
728
views
Simulate power loss with force unmount?
I want to test disaster recovery of RDBMs after power loss under high load.
My idea is to mount data directory under new mountpoint and then execute umount -f during the load and investigate outcome /...
- 248