User Tools

Site Tools


linux:virtualization:lxc

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
linux:virtualization:lxc [2016/08/16 10:47]
tkilla [IPv6 setup]
linux:virtualization:lxc [2022/01/13 23:06]
tkilla [Create new container]
Line 86: Line 86:
 **reload:** **reload:**
   sysctl -p /etc/sysctl.conf     sysctl -p /etc/sysctl.conf  
 +
 +
 +
 +
 +
 +\\
 +==== Simple Nat Bridge ====
 +
 +Easy version without libvirt - work well at OVH and hetzner. 
 +
 +**Add an additional bridge (keep eth0 as is) in /etc/network/interfaces**
 +
 +  auto lxc-bridge
 +  iface lxc-bridge inet static
 +        bridge_ports none
 +        bridge_fd 0
 +        bridge_maxwait 0
 +        address 192.168.10.1
 +        netmask 255.255.255.0
 +        up iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
 +
 +
 +**activate forwarding temporary:**
 +  echo 1 > /proc/sys/net/ipv4/ip_forward
 +
 +**activate forwarding permanent:**
 +
 +Uncomment in /etc/sysctl.conf
 +  net.ipv4.ip_forward=1
 +
 +Activate new settings:
 +  sysctl -p
 +
 +
 +**firewall rules**
 +
 +  # intern -> extern
 +  iptables -t nat -A POSTROUTING -s 192.168.10.10/24 -j SNAT --to-source 1.2.3.4
 +
 +  # ports extern -> intern - 1 rule for each $PORT
 +  iptables -t nat -A PREROUTING  -d 1.2.3.4 -p tcp --dport $PORT -j DNAT --to 192.168.10.10:${PORT}
 +
 +
 +**Container config:**
 +  lxc.network.type = veth
 +  lxc.network.flags = up
 +  lxc.network.link = lxc-bridge
 +  lxc.network.ipv4.gateway = 192.168.10.1
 +  lxc.network.ipv4 = 192.168.10.10/24
 +
 +
 +
 +
 +  * https://wiki.debian.org/LXC/SimpleBridge
 +
  
  
 \\ \\
 ==== Nat Bridge via libvirt and IPV6 ==== ==== Nat Bridge via libvirt and IPV6 ====
 +
 +**Deprecated! - if you not need DHCP, use the simple bridge method without libvirt/virsh!**
  
   apt-get install libvirt-bin   apt-get install libvirt-bin
Line 119: Line 176:
     </ip>     </ip>
     <!-- IPV6 :: -->     <!-- IPV6 :: -->
-   <ip family='ipv6' address='2001:41d0:2:bb10::2' prefix='64'>+   <ip family='ipv6' address='1234:1234:1234:1234::2' prefix='64'>
    </ip>         </ip>     
   </network>   </network>
Line 128: Line 185:
   lxc.network.flags = up   lxc.network.flags = up
   lxc.network.link = virbr0   lxc.network.link = virbr0
-  lxc.network.hwaddr = CC:AA:FF:EE:00:01   # not required+  lxc.network.hwaddr = CC:AA:FF:EE:00:01   # not required and not useful: inter-vs-connection breaks whene set
   lxc.network.ipv4 = 192.168.122.100/24   lxc.network.ipv4 = 192.168.122.100/24
   lxc.network.ipv4.gateway = auto   # auto usually works, otherwise set main IP gateway (.254 at OVH)   lxc.network.ipv4.gateway = auto   # auto usually works, otherwise set main IP gateway (.254 at OVH)
  
   # ipv6:   # ipv6:
-  lxc.network.ipv6 = 2001:41d0:0002:bb10:0000:0000:0000:0100/64+  lxc.network.ipv6 = 1234:1234:1234:1234:0100/64
   lxc.network.ipv6.gateway = auto   lxc.network.ipv6.gateway = auto
  
Line 151: Line 208:
   ## IPv6:   ## IPv6:
   iface eth0 inet6 static   iface eth0 inet6 static
-    address 2001:41d0:2:bb10::131+    address 1234:1234:1234:1234::131
     netmask 64     netmask 64
  
Line 167: Line 224:
   virsh net-destroy default   virsh net-destroy default
   virsh net-start default   virsh net-start default
 +
 +**if you remove this net, disable autostart**
 +  virsh net-autostart default --disable
 +
  
 **activate forwarding temporary:** **activate forwarding temporary:**
Line 175: Line 236:
 Uncomment in /etc/sysctl.conf Uncomment in /etc/sysctl.conf
   net.ipv4.ip_forward=1   net.ipv4.ip_forward=1
 +
 +Activate new settings:
 +  sysctl -p
  
 **iptables config ( 1.2.3.4 is pubilc ip in root):** **iptables config ( 1.2.3.4 is pubilc ip in root):**
Line 200: Line 264:
  
   echo "IPv6 setup FOREACH vserver.."   echo "IPv6 setup FOREACH vserver.."
-  ip -6 route add 2001:41d0:2:bb10::100 dev virbr0 +  ip -6 route add 1234:1234:1234:1234::100 dev virbr0 
-  ip -6 neigh add proxy 2001:41d0:2:bb10::100 dev eth0 +  ip -6 neigh add proxy 1234:1234:1234:1234::100 dev eth0 
-  ping6 -I virbr0 -c 5 2001:41d0:2:bb10::100+  ping6 -I virbr0 -c 5 1234:1234:1234:1234::100
  
 \\ \\
Line 277: Line 341:
 ==== Create new container ==== ==== Create new container ====
  
-  lxc-create -n websrv -t debian-wheezy  -B btrfs+for unprivileged containers, edit /etc/subuid and /etc/subgid first and add matching lines to /etc/lxc/default.conf - see below! 
 +http://wiki.fr33.info/doku.php/linux/virtualization/lxc?&#unprivileged_containers 
 + 
 +FIX:  original keyserver is broken! add: --keyserver hkp://keyserver.ubuntu.com 
 + 
 + 
 +  lxc-create -n debian8  -B btrfs -t debian -- -r jessie --keyserver hkp://keyserver.ubuntu.com 
 + 
 +or 
 + 
 +  lxc-create -n websrv -t debian-wheezy  -B btrfs --keyserver hkp://keyserver.ubuntu.com
  
 Start / Stop VS: Start / Stop VS:
Line 285: Line 359:
 Enter VS: Enter VS:
   lxc-console -n websrv   lxc-console -n websrv
 +
 +
 +In Buster, use the lxc-download script:
 +
 +  /usr/share/lxc/templates/lxc-download --list --no-validate| grep debian | grep amd64
 +  lxc-create -t /usr/share/lxc/templates/lxc-download -n <NAME> --  --no-validate -d debian -r buster -a amd64
  
  
Line 295: Line 375:
   lxc-clone --backingstore btrfs --orig vs1 --new vs2 --snapshot   lxc-clone --backingstore btrfs --orig vs1 --new vs2 --snapshot
  
 +\\
 +===== Mount external Dirs in Container =====
 +
 +The recommended way is to add the mountpoint with a relative path in the VS config:
 +
 +  lxc.mount.entry=/home/mountme home none bind,optional,relative,create=dir
 +
 +
 +Under some cicumstances it does not work (in unprivileged containers), but this works:
 +
 +  lxc.mount.entry = /home/test /home/vservers/stretch/rootfs/home/test none bind 0 0
 +
 +Also check Permissions and Ownership. chown to the root ID inside the container.
  
 \\ \\
 ===== brtfs snapshots ===== ===== brtfs snapshots =====
 +
 +the container must be stopped for a lxc-snapshot. use btrfs snapshot to backup running containers (mysql may get inconsitent)
  
 you need to create container with option  -B btrfs!! you need to create container with option  -B btrfs!!
  
   lxc-create -B btrfs -n mycontainer -t ubuntu   lxc-create -B btrfs -n mycontainer -t ubuntu
 +
 +
  
  
Line 307: Line 404:
  
   mv /home/vservers/my-lxc-container/rootfs /home/vservers/my-lxc-container/rootfs.saved   mv /home/vservers/my-lxc-container/rootfs /home/vservers/my-lxc-container/rootfs.saved
-  btrfs subvolume create /home/vservers/my-lxc-container/rootfs+  btrfs subvolume create /home/vservers/my-lxc-container/rootfs  
   btrfs subvolume list /home/vservers   btrfs subvolume list /home/vservers
-  lxc-snapshot -n webdev 
      
   # for unprivileged root container, check UID and GID of rootfs dir (here it is 100000):   # for unprivileged root container, check UID and GID of rootfs dir (here it is 100000):
   chown 100000:100000 /home/vservers/webdev/rootfs/   chown 100000:100000 /home/vservers/webdev/rootfs/
  
 +  mv /home/vservers/my-lxc-container/rootfs.saved/* /home/vservers/my-lxc-container/rootfs/
 +  lxc-snapshot -n webdev
 +
 +snapshot with comment
 +
 +  echo "working my-lxc-container before ..." > snap-comment
 +  lxc-stop -n my-lxc-container
 +  lxc-snapshot -n my-lxc-container -c snap-comment
 +  rm snap-comment
  
   * https://uli-heller.github.io/blog/2013/06/09/lxc-snapshots/   * https://uli-heller.github.io/blog/2013/06/09/lxc-snapshots/
 +
 +
 +==== Copy container ====
 +
 +To move the container to another machine, .. - take care of the user/group IDs:
 +
 +pack:
 +  tar --numeric-owner -czvf container.tar.gz ./*
 +
 +move..
 +
 +unpack:
 +  tar --numeric-owner -xzvf container.tar.gz ./*
 +
 +
 \\ \\
 ===== Security ===== ===== Security =====
  
 ==== Unprivileged containers ==== ==== Unprivileged containers ====
 +
 +uids and gids are shifted to another scope. so root uid 0 becomes 100000 for example. inside the container this is not visible, but from outside you can see the uid 100000+. you can still run these containers as root. you only have to add root in /etc/subuid and /etc/subgid - than its the same as running the containers as user.
 +
 +for best security, each container should have its own uid/gid space, although it is unlikely to break out of one container and enter another.
 +
  
 only available in > v1.0, not in debian squeeze :( only available in > v1.0, not in debian squeeze :(
  
-run unprivileged container as root:+**run unprivileged container as root:**
  
 add root to /etc/subuid and /etc/subgid, add root to /etc/subuid and /etc/subgid,
   root:100000:65536   root:100000:65536
  
-vs config - map user ids:+ 
 +**vs config - map user ids:** 
 + 
 +put this in /etc/lxc/default.conf too! 
   lxc.id_map = u 0 100000 65536   lxc.id_map = u 0 100000 65536
   lxc.id_map = g 0 100000 65536   lxc.id_map = g 0 100000 65536
  
 +in buster it's called idmap:
 +  lxc.idmap = u 0 100000 65536
 +  lxc.idmap = g 0 100000 65536
  
-create container - use download method. jessie is not available, so you can upgrade wheezy and fix systemd error :(+**shift uuids to another span:** 
 + 
 +Use this script: https://github.com/exaexa/chownmap 
 + 
 +  /root/bin/chownmap 0 200000 65536 /home/vservers/<containername>/rootfs/ 
 + 
 + 
 + 
 +create container - use download method for unprivileged. jessie is not available, so you can upgrade wheezy and fix systemd error :(
  
   lxc-create -B btrfs -t download -n websrv      lxc-create -B btrfs -t download -n websrv   
Line 343: Line 483:
   LANG=C SUITE=jessie MIRROR=http://httpredir.debian.org/debian lxc-create -n websrv -B btrfs -t debian   LANG=C SUITE=jessie MIRROR=http://httpredir.debian.org/debian lxc-create -n websrv -B btrfs -t debian
  
-.... 
  
 +
 +**Unprivileged related bugfixes**
 +
 +  lxc-start: Permission denied - failed to create directory '/usr/lib/x86_64-linux-gnu/lxc/rootfs/lxc_putold'
 +
 +Is caused by wrong permissions of rootfs. Set:
 +
 +  chown 100000:100000 /vservers/<containername>/rootfs
 +
 +If you copy files from outside into the container, they have wrong uid/gid. if the file should belong to root, just run this from the root system:
 +
 +  chown 100000:100000 /vservers/<containername>/rootfs/<path to the file in container>
 +
 +
 +
 +**Links:**
   * https://unix.stackexchange.com/questions/127554/building-unprivileged-userns-lxc-container-from-scratch-by-migrating-a-privil   * https://unix.stackexchange.com/questions/127554/building-unprivileged-userns-lxc-container-from-scratch-by-migrating-a-privil
   * https://www.stgraber.org/2014/01/01/lxc-1-0-security-features/   * https://www.stgraber.org/2014/01/01/lxc-1-0-security-features/
   * https://blog.deimos.fr/2014/08/29/lxc-1-0-on-debian-wheezy/   * https://blog.deimos.fr/2014/08/29/lxc-1-0-on-debian-wheezy/
   * https://wiki.deimos.fr/LXC_:_Install_and_configure_the_Linux_Containers   * https://wiki.deimos.fr/LXC_:_Install_and_configure_the_Linux_Containers
 +  * https://unix.stackexchange.com/questions/177030/what-is-an-unprivileged-lxc-container
  
  
Line 404: Line 560:
  
   * https://github.com/debops/ansible-lxc/issues/15   * https://github.com/debops/ansible-lxc/issues/15
 +
 +** systemd cgroups fuckup**
 +
 +Could not find writable mount point for cgroup hierarchy 12 while trying to create cgroup
 +
 +12 is a systemd hierarchy - if you remove systemd and switch to sysvinit-core, this might be leftover.
 +
 +FIXME:
 +
 +check all of systemd is gone (uninstall ii):
 +  dpkg -l *systemd*
 +  apt remove --purge *systemd*    # without systemd apt/preferences.d/ must not be set
 +  
 +/etc/pam.d/common-session - unset this line:
 +
 +  session     optional    pam_cgfs.so -c freezer,memory,name=systemd
 +
 +Check, if 12 is still active:
 +
 +  cat /proc/self/cgroup
 +
 +WORKAROUND:
 +mcedit /etc/lxc/lxc.conf and remove 
 +  lxc.cgroup.use = @all
 +
 +* this is helpful: https://github.com/lxc/lxc/issues/1279
 +* this is not: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769494
 +
 +\\
  
 **SSH Config** **SSH Config**
Line 420: Line 605:
  
   * http://gaijin-nippon.blogspot.de/2013/07/audit-on-lxc-host.html   * http://gaijin-nippon.blogspot.de/2013/07/audit-on-lxc-host.html
 +
 +or run this inside the container:
 +
 +  sed '/pam_loginuid.so/s/^/#/g' -i /etc/pam.d/
  
 FIXME maybe insecure FIXME maybe insecure
Line 432: Line 621:
  
  
 +**rsyslog error**
 +
 +TESTME
 +
 +rsyslog doesnt start on boot and errors in syslog:
 +  .. rsyslogd: imklog: cannot open kernel log(/proc/kmsg): Permission denied.
 +  .. rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
 +
 +Disable kernel logging in container /etc/rsyslog.conf:
 +
 +  # $ModLoad imklog   # provides kernel logging support
 +
 +
 +
 +\\
 +===== Move a root system into container =====
 +
 +you can disable many services:
 +  * udev: udev service (which is a hard dependency of systemd in Jessie) won't run in a container, but systemd recognized it
 +  * apparmor: mounts security-fs, would need to disable drop caps. bad idea
 +  * kmod
 +  * lm-sensors
 +  * dbus
 +  * kbd
 +  * hdparm
 +  * ...
 +
 +configuration changes
 +
 +  * various sysctl.conf options do not work inside container
 +  * /etc/modules -loading do not work
 + 
 +If you get errors like:
 +  INIT: Id "6" respawning too fast: disabled for 5 minutes 
 +
 +disable the matching line in /etc/inittab:
 +  # 5:23:respawn:/sbin/getty 38400 tty5
  
linux/virtualization/lxc.txt · Last modified: 2022/01/13 23:08 by tkilla