Sidebar
linux:virtualization:lxc
This is an old revision of the document!
Table of Contents
LXC
Installation
Networking
Options:
- Create a bridge on the host (natted/routed),
- VLAN + bridge setup description,
- Use libvirt package for easy network setup
Network resources:
- Working bridge config (french) http://blog.champs-libres.coop/sysadmin/2015/08/18/ip-failover-ovh-container-lxc.html
- Nat and bridge howto - manual bridge adding may result in network loss after 30-60min https://stackoverflow.com/questions/25042542/how-do-i-connect-a-lxc-container-to-an-ip-alias?rq=1
Bridge
Exposes the public IPs into a container.
Setup for OVh / SoYouStart failover IPs
The failover IPs can be mapped directly into containers.
A Virtual Mac Address must be assigned to the failover IP in ovh webmanager first!
Check for too many ARP requests from the container! OVH may block the IP. The requests can be avoided by assigning Virtual MACs to all failover IPs:
tcpdump -varp
Do not setup the failover IPs in /etc/network/interfaces on host or in vs - only in the container config file!
Move eth0 to br0 in /etc/network/interfaces - eth0 becomes “static” and is added as “bridge_ports eth0”:
auto eth0
iface eth0 inet manual
auto br0
iface br0 inet static
address 1.2.3.4
netmask 255.255.255.0
network 1.2.3.0
broadcast 1.2.3.255
gateway 1.2.3.254 # at ovh the gateway is always the main ip's x.x.x.254
bridge_ports eth0 # add these devices to bridge (can be eth0 eth1 ..)
bridge_stp off # no spanning tree protocol
bridge_fd 0
bridge_maxwait 0
Container config:
# comment the default empty network interface #lxc.network.type = empty
# bridge setup: lxc.network.type = veth # lxc network type (an additional virtual bridge will be created) lxc.network.flags = up # start on vs boot lxc.network.link = br0 # host bridge iface lxc.network.ipv4 = fail.over.ip.x/24 # failover ip to use / 24 lxc.network.ipv4.gateway = MAIN.IP.GATEWAY.254 # gateway of the main ip is x.x.x.254 lxc.network.hwaddr = 02:00:00:3c:95:31 # IMPORTANT! the MAC address assigned to the failover IP in webmanager
Inside VS /etc/network/interfaces:
auto lo iface lo inet loopback
auto eth0 iface eth0 inet manual
#auto eth0 #iface eth0 inet dhcp
Setup bridge options in /etc/sysctl.conf:
net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0
reload:
sysctl -p /etc/sysctl.conf
Nat Bridge via libvirt and IPV6
apt-get install libvirt-bin virsh net-info default virsh net-start default
autostart:
virsh net-autostart default virsh net-info default
configure - do not use editor - its overwritten! - set static ips:
first set mcedit as default editor in /root/.bashrc:
export EDITOR='mcedit'
virsh net-edit default
<network>
<name>default</name>
<uuid>a07016bb-2e96-2000-9e16-b93b12245329</uuid>
<forward mode='nat'/>
<bridge name='virbr0' stp='on' delay='0' />
<mac address='51:51:00:FC:01:E1'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
<host mac='CC:AA:FF:EE:00:01' name='websrv.somedomain.com' ip='192.168.122.100' />
</dhcp>
</ip>
<!-- IPV6 :: -->
<ip family='ipv6' address='2001:41d0:2:bb10::2' prefix='64'>
</ip>
</network>
edit VS config, add:
lxc.network.type = veth lxc.network.flags = up lxc.network.link = virbr0 lxc.network.hwaddr = CC:AA:FF:EE:00:01 # not required lxc.network.ipv4 = 192.168.122.100/24 lxc.network.ipv4.gateway = auto # auto usually works, otherwise set main IP gateway (.254 at OVH)
# ipv6: lxc.network.ipv6 = 2001:41d0:0002:bb10:0000:0000:0000:0100/64 lxc.network.ipv6.gateway = auto
edit /etc/network/interfaces _inside_ VS - just set eth0 to manual:
# The loopback network interface auto lo iface lo inet loopback iface lo inet6 loopback
## IPv4: auto eth0 iface eth0 inet manual
## IPv6: iface eth0 inet6 static address 2001:41d0:2:bb10::131 netmask 64
Check /etc/resolv.conf inside VS:
# for testing, add google's dns: nameserver 8.8.8.8
make persistent:
virsh net-autostart default
restart network:
virsh net-destroy default virsh net-start default
activate forwarding temporary:
echo 1 > /proc/sys/net/ipv4/ip_forward
activate forwarding permanently:
Uncomment in /etc/sysctl.conf
net.ipv4.ip_forward=1
iptables config ( 1.2.3.4 is pubilc ip in root):
iptables -t nat -I PREROUTING -p tcp -d 1.2.3.4 --dport 80 -j DNAT --to-destination 192.168.122.100:80 iptables -I FORWARD -m state -d 192.168.122.100/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
ip6tables config for IPv6:
echo "generic IPv6 setup.."
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT # NEW or not? NEW, ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ip6tables -t filter -A INPUT -i lo -j ACCEPT ip6tables -t filter -A OUTPUT -o lo -j ACCEPT ip6tables -t filter -P FORWARD ACCEPT
# ping ip6tables -t filter -A INPUT -p ipv6-icmp -j ACCEPT ip6tables -t filter -A OUTPUT -p ipv6-icmp -j ACCEPT
echo "IPv6 setup FOREACH vserver.." ip -6 route add 2001:41d0:2:bb10::100 dev virbr0 ip -6 neigh add proxy 2001:41d0:2:bb10::100 dev eth0 ping6 -I virbr0 -c 5 2001:41d0:2:bb10::100
IPv6 setup
.. is tricky!
besides the settings in the previous section (virsh, iptables, interfaces,..) it may be required to setup:
/etc/sysctl.conf - working setting
# v2 - vs ipv6: net.ipv6.conf.all.autoconf = 0 net.ipv6.conf.default.autoconf = 0 net.ipv6.conf.eth0.autoconf = 0 net.ipv6.conf.default.accept_ra = 0 # accept_ra = 2 seems a bit weird for a boolean. It's a special value to allow IPv6 forwarding AND RA net.ipv6.conf.all.accept_ra = 2 net.ipv6.conf.eth0.accept_ra = 2
# bridging (tk) net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 # reuqired for ipv6 forwarding net.ipv6.conf.all.proxy_ndp = 1 #net.ipv6.conf.eth0.proxy_ndp = 1 #net.ipv6.conf.virbr0.proxy_ndp=1 net.ipv6.conf.default.forwarding = 1
radvd neighbouring deamon seems to be required, too: /etc/radvd.conf
interface virbr0
{
AdvReachableTime 300000;
# MinRtrAdvInterval 60;
# MaxRtrAdvInterval 120;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvSendAdvert on;
AdvManagedFlag off;
AdvOtherConfigFlag off;
prefix <IPv6 here>::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
}
Helpful commands - inside and outside of container:
ip -6 r ip neigh show
ping6 container_ip ping6 world_from_container
- often ipv6 starts to work, when you ping sometimes in ↔ out → the neighbouring needs to be build up and stales after a short time
- include ping6 in firewall startup builds up the route
- something about “proxy” is required - see ip6tables above
Usage
Create new container
for unprivileged containers, edit /etc/subuid and /etc/subgid first and add matching lines to /etc/lxc/default.conf - see below! http://wiki.fr33.info/doku.php/linux/virtualization/lxc?&#unprivileged_containers
lxc-create -n debian8 -B btrfs -t debian -- -r jessie
or
lxc-create -n websrv -t debian-wheezy -B btrfs
Start / Stop VS:
lxc-start -n websrv lxc-stop -n websrv
Enter VS:
lxc-console -n websrv
Clone container
Copy all data:
lxc-clone --backingstore btrfs --orig vs1 --new vs2
Make a brtfs snapshot:
lxc-clone --backingstore btrfs --orig vs1 --new vs2 --snapshot
brtfs snapshots
you need to create container with option -B btrfs!!
lxc-create -B btrfs -n mycontainer -t ubuntu
workaround to add btrfs snapshots after creating container
mv /home/vservers/my-lxc-container/rootfs /home/vservers/my-lxc-container/rootfs.saved btrfs subvolume create /home/vservers/my-lxc-container/rootfs btrfs subvolume list /home/vservers lxc-snapshot -n webdev # for unprivileged root container, check UID and GID of rootfs dir (here it is 100000): chown 100000:100000 /home/vservers/webdev/rootfs/
Copy container
To move the container to another machine, .. - take care of the user/group IDs:
pack:
tar --numeric-owner -czvf container.tar.gz ./*
move..
unpack:
tar --numeric-owner -xzvf container.tar.gz ./*
Security
Unprivileged containers
only available in > v1.0, not in debian squeeze :(
run unprivileged container as root:
add root to /etc/subuid and /etc/subgid,
root:100000:65536
vs config - map user ids:
put this in /etc/lxc/default.conf too!
lxc.id_map = u 0 100000 65536 lxc.id_map = g 0 100000 65536
create container - use download method. jessie is not available, so you can upgrade wheezy and fix systemd error :(
lxc-create -B btrfs -t download -n websrv
# error no jessie: lxc-create -B btrfs -n websrv -t download -- -d debian -r jessie -a amd64
# error not working with unprivileged LANG=C SUITE=jessie MIRROR=http://httpredir.debian.org/debian lxc-create -n websrv -B btrfs -t debian
Bugfixes:
lxc-start: Permission denied - failed to create directory '/usr/lib/x86_64-linux-gnu/lxc/rootfs/lxc_putold'
Is caused by wrong permissions of rootfs. Set:
chown 100000:100000 /vservers/<containername>/rootfs
Links:
Bugfixes
- If a container doesn't start, check if /dev /sys /proc exist
- Check log files in /containerrootdir/
- Check network settings in config
Network - no outgoing connection from container:
- check resolv.conf
- reboot: iptables may be messed up
Squeeze-LTS Containers are tricky:
- If they don't start, use /etc/ from a default container.
- If you cannot login “no more processes in this runlevel” use /etc/inittab from default container. The habe a different tty setup:
#5:23:respawn:/sbin/getty 38400 tty5 #6:23:respawn:/sbin/getty 38400 tty6
- If you get “TERM variable not set”, set it, e.g. in /root/.bashrc:
TERM=linux export TERM
Jessie Containers are tricky:
systemd prevents start.
remove systemd or prevent install before upgrade to jessie:
fix by adding this to container config - didnt always work, sometimes dbus errors appear when apt'ing
# Custom container options lxc.mount.auto = cgroup:mixed lxc.mount.entry = tmpfs dev/shm tmpfs rw,nosuid,nodev,create=dir 0 0 lxc.mount.entry = tmpfs run tmpfs rw,nosuid,nodev,mode=755,create=dir 0 0 lxc.mount.entry = tmpfs run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,create=dir 0 0 # lxc.mount.entry = debugfs sys/kernel/debug debugfs rw,relatime 0 0 lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir 0 0 # lxc.mount.entry = hugetlbfs dev/hugepages hugetlbfs rw,relatime,create=dir 0 0
Also make sure, that you have the following line in your /etc/lxc/lxc.conf:
lxc.cgroup.use = @all
SSH Config
root login Debian Jessie default container has this option set, so you cannot login with password as root:
PermitRootLogin without-password
pam login error
error: PAM: pam_open_session(): Cannot make/remove an entry for the specified session
/etc/pam.d/sshd
# session required pam_loginuid.so
maybe insecure
apt-get update broken via ipv6
apt-get update -o Acquire::ForceIPv4=true
permanent apt via ipv4:
echo 'Acquire::ForceIPv4 "true";' | tee /etc/apt/apt.conf.d/99force-ipv4
linux/virtualization/lxc.1475159351.txt.gz · Last modified: 2016/09/29 16:29 by tkilla
