====== Crash Recovery ======

===== Strato Root Server =====


Login to Remote-Console first, so you can see boot messages, ... and kernel panics, if unlucky

Re-Install takes 1h

==== Recovery Console: ====


Set Boot mode to recovery in "RecoveryManager" at https://config.stratoserver.net/
Wait for 15-20min!! until "ServerData" Serverstatus shows: Installation finished

Strato sets RAIDs to /md/126 and md127 - **that sucks!** - via grub install, ...,.. you can mess up the system

log in to recovery via 

  ssh root@IP   # recovery root passwd is shown in "Serverdata"
  
  mount /dev/md127 /mnt/    # system partition
  mount /dev/md126 /mnt/boot/    # boot partition

to re-install grub, you need to chroot

  mount -o bind /dev /mnt/dev
  mount -t proc none  /mnt/proc/
  mount -t sysfs none /mnt/sys

  chroot /mnt
  update-grub
  
  ( grub-mkdevicemap  #  In case of error “no such disc” )

  grub-install /dev/sda
  grub-install /dev/sdb

To exit the recovery console you need to set boot mode to normal and wait 10min until "ServerData" Serverstatus shows: Installation finished

===== SoYouStart / OVH Root Server =====

**You need to setup your ssh key in the webmanager to enter in rescue mode without password!** the password will be sent by mail as well.

- Change netboot mode in web-manager: set to "rescue"
- Click "reboot"

Mount partitions:
  mkdir /mnt/md2
  mount /dev/md2 /mnt/md2/

After restore, click "netboot" in webmanager again and reset it to "harddisc"



===== Full Disc Encrypted System Rescue =====

If the system does not boot anymore, **you can open the encrypted device anyway with a live system** on USB stick or CD. 

  # find the correct partitions:
  fdisk -l 
  
  # open the container 
  # "sdb5_crypt" is just an example. **Check what's written in /etc/crypttab - the names must match!**
  cryptsetup luksOpen /dev/sdXY sdXY_sdb5_crypt
  
  # activate the LVM volume group (if you use LVM2):
  vgchange -ay
  
  # find new volume groups in /dev/mapper/ and mount them (maybe it's just root or home, var, ...might be in, too):
  mount /dev/mapper/ubuntu--vg-root /mnt
  
  # mount the unencrypted boot parti:
  mount /dev/sdXZ /mnt/boot
  
  # bind mount some important dirs:
  mount -t proc proc /mnt/proc
  mount -o bind /dev /mnt/dev
  mount -t sysfs none /mnt/sys


Now you can take your data back already.

To repair what is broken, you probably need to change root into that system:

  chroot /mnt

All following commands are executed inside the broken system. Some hints for repairs:

  apt install lvm2 cryptsetup-initramfs    # debian uninstalled lvm during an upgrade :(

If your root system is BTRFS, you need btrfs-progs and the kernel module during the early initramfs boot stage:

  apt install btrfs-progs


Add a missing module to initramfs - inside the chroot:

  mcedit /etc/initramfs-tools/modules


Write one module name like "btrfs" per line.

Then regenerate the initramfs for all installed kernels:

  update-initramfs -u -k all


* Check that the name in /etc/crypttab is the same as in the cryptsetup luksOpen Command, usually something like sda5_crypt
* Also check the UUID entries in /etc/crypttab compared to ls -la /dev/disk/by-uuid/


TbC..  There may be various other reasons for bootproblems...



== Links: ==

  * really helpul! https://feeding.cloud.geek.nz/posts/recovering-from-unbootable-ubuntu-encrypted-lvm-root-partition/
  * various other hints https://askubuntu.com/questions/567730/gave-up-waiting-for-root-device-ubuntu-vg-root-doesnt-exist#567897