bikinibottom/TuringPi/docker-swarm.md

Docker Swarm

1. Compute Modules

Install OS / headless installation

1. Flash the image to a SD card/ EMMC.
2. Go to Boot partition.
3. Then change the DTB listed in /boot/extlinux/extlinux.conf to -model-a.dtb instead of -cm4.dtb.
4. Now put the SD card/ EMMC to the preferred base board and boot.
5. First boot will take time as the user partition will resize to take whole storage size.

ssh as "root" and complete setup

sudo apt update && sudo apt upgrade

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

sudo apt install -y docker-compose docker git vim screenfetch net-tools wget curl

sudo apt autoremove

Edit

sudo vim /etc/sudoers.d/pleb

pleb ALL=(ALL:ALL) NOPASSWD: ALL

Set Static IP

Create/Edit

sudo vim /etc/systemd/network/end0.network

[Match]
Name=end0

[Network]
Address=192.168.1.xxx/24
Gateway=192.168.1.254
# OpenDNS
DNS=208.67.222.222
DNS=208.67.220.220

Edit

sudo vim /etc/hosts

# Host addresses
127.0.0.1  localhost
127.0.1.1  sandy
# Nodes
192.168.1.14 sandy   sandy.local 
192.168.1.13 gary    gary.local
192.168.1.16 sheldon sheldon.local
192.168.1.19 pearl   pearl.local

Reboot

On remote :

ssh-keygen -t ed25519
ssh-copy-id -f -i ~/.ssh/id_ed25519.pub pleb@<host>
ssh pleb@<host>

Edit

sudo vim /etc/ssh/sshd_config

PasswordAuthentication no

Restart service

sudo systemctl restart sshd

---

sudo systemctl enable iptables

sudo systemctl start iptables

sudo iptables -I INPUT -p tcp -m tcp --dport 2376 -j ACCEPT
sudo iptables -I INPUT -p tcp -m tcp --dport 2377 -j ACCEPT
sudo iptables -I INPUT -p tcp -m tcp --dport 7946 -j ACCEPT
sudo iptables -I INPUT -p udp -m udp --dport 7946 -j ACCEPT
sudo iptables -I INPUT -p tcp -m udp --dport 4789 -j ACCEPT
sudo iptables -I INPUT -p 50 -j ACCEPT # allows ipsec when secure overlay is enabled

Restart...

---

2. Docker Swarm

Install docker...

sudo apt install -y docker docker-compose git

sudo usermod -aG docker pleb

sudo systemctl enable docker.service
sudo systemctl start docker.service

sudo chmod 666 /var/run/docker.sock

---

On master

docker swarm init --advertise-addr 192.168.1.14

Swarm initialized: current node (z76p66jcv4c1c9qstodlpcfii) is now a manager.

On nodes

docker swarm join --token SWMTKN-1-27tm53ylrwyesiq3ntfumfixw3gpe2164rnsume8sg07saz12v-1fig1jzehaocodm6hv322l4ku 192.168.1.14:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Change ip range if necessary ?!

docker network rm ingress
# Create in different range
docker network create --driver overlay --ingress --subnet 192.168.1.0/16 --gateway 192.168.1.254 ingress

---

docker node ls

ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
lqu5nr20g6vt2g1hc5xwm2njg     gary       Ready     Active                          29.6.1
3aqxc8mj1wk2qp1k0llodgyxw     pearl      Ready     Active                          29.6.1
z76p66jcv4c1c9qstodlpcfii *   sandy      Ready     Active         Leader           29.6.1
i62rvg0fh6c0hv49btb1qrfv4     sheldon    Ready     Active                          29.6.1

docker network ls

NETWORK ID     NAME              DRIVER    SCOPE
3b8a522fd1ad   bridge            bridge    local
8624696f7cbd   docker_gwbridge   bridge    local
116b95142a3a   host              host      local
jg83q53aljof   ingress           overlay   swarm
46f3064ef5dc   none              null      local

---

3. Keepalived

Load Balancer : install on all nodes

sudo apt install keepalived -y

On master node (sandy)

Create/Edit

sudo vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@delmar.bzh
   }
   notification_email_from admin@delmar.bzh
   smtp_server pro1.mail.ovh.net
   smtp_connect_timeout 30
   router_id docker_ingress
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface end0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass transatlantique
    }
    virtual_ipaddress {
        192.168.1.21
    }
}

On Nodes

Create/Edit

sudo vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@delmar.bzh
   }
   notification_email_from admin@delmar.bzh
   smtp_server pro1.mail.ovh.net
   smtp_connect_timeout 30
   router_id docker_ingress
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface end0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass transatlantique
    }
    virtual_ipaddress {
        192.168.1.21
    }
}

Start/Enable Service

sudo systemctl enable keepalived
sudo systemctl start keepalived

ip a show end0

---

4. Glusterfs

Distributed Storage : install on all nodes

sudo apt-get -y install glusterfs-server

sudo systemctl enable glusterd
sudo systemctl start glusterd

sudo fdisk -l

# Format your disk (If you decided to use your internal storage, just ignore the mount and format steps)
sudo mkfs.xfs -f /dev/nvme0n1

# Create brick folder location
sudo mkdir -p /mnt/ssd

# Add line to /etc/fstab to auto mount this disk to /data on boot
echo "UUID=... /mnt/ssd xfs defaults 0 0" | sudo tee -a /etc/fstab

# Create brick folder
sudo mkdir -p /mnt/ssd/brick

# Mount
sudo mount -a

sudo systemctl daemon-reload

sudo gluster peer probe gary
sudo gluster peer probe sheldon
sudo gluster peer probe pearl

Check

df -h /mnt/ssd/brick

sudo gluster pool list

UUID                                    Hostname        State
a40ae0e1-90ec-4ccb-88eb-0a13c6c0de26    gary            Connected 
fc215f6f-38df-4cce-a0d3-5297e0290d3f    sheldon         Connected 
13233744-1367-4e76-b486-a0e0b8a190e7    pearl           Connected 
07702514-efbc-4ca6-950d-d9c0b14ff61c    localhost       Connected

sudo gluster peer status

Number of Peers: 3

Hostname: gary
Uuid: a40ae0e1-90ec-4ccb-88eb-0a13c6c0de26
State: Peer in Cluster (Connected)

Hostname: sheldon
Uuid: fc215f6f-38df-4cce-a0d3-5297e0290d3f
State: Peer in Cluster (Connected)

Hostname: pearl
Uuid: 13233744-1367-4e76-b486-a0e0b8a190e7
State: Peer in Cluster (Connected)


> For 2 nodes: sandy and sheldon for example

```bash
sudo gluster volume create docker-volume replica 2 transport tcp sandy:/mnt/ssd sheldon:/mnt/ssd force

For all nodes: sandy, gary, sheldon and pearl for example

sudo gluster volume create dockervolume disperse 4 redundancy 1 sandy:/mnt/ssd/brick gary:/mnt/ssd/brick sheldon:/mnt/ssd/brick pearl:/mnt/ssd/brick force

Start Gluster Volume

sudo gluster volume info dockervolume

sudo gluster volume start dockervolume

On all nodes :

sudo mkdir /mnt/ssd

echo "localhost:/dockervolume /mnt/ssd/brick glusterfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab

sudo mount -a

sudo systemctl daemon-reload

Check

Persistent shared storage will be "/mnt/ssd/brick"

df -h /mnt/ssd/brick

sudo gluster volume set dockervolume auth.allow 127.0.0.1,192.168.1.14,192.168.1.13,192.168.1.16,192.168.1.19

When adding a new brick:

sudo gluster volume add-brick **volume** replica X **server:brick**

Option

docker plugin install --grant-all-permissions \
  trajano/glusterfs-volume-plugin

Use in a stack file

services:
  app:
    image: myapp/web:latest
    volumes:
      - app_data:/app/data

volumes:
  app_data:
    driver: trajano/glusterfs-volume-plugin
    driver_opts:
      glusteropts: "--volfile-server=node-01 --volfile-id=swarm-vol"
      subdir: "app_data"

5. Portainer

Create container directory

sudo mkdir -p /var/lib/docker/volumes

sudo mkdir -p /mnt/ssd/brick/portainer

Edit

sudo vim portainer-agent-stack.yml

Down at the bottom, remove :

volumes:
  portainer_data:
  

And in Section services -> portainer -> volumes, change it to :

    volumes:
      - type: bind
        source: /mnt/ssd/brick/portainer/portainer_data
        target: /data
   

Deploy

docker stack deploy -c portainer-agent-stack.yml portainer

Upgrade

docker service ls

docker pull portainer/portainer-ce:latest

docker service update --image portainer/portainer-ce:latest --publish-add 9443:9443 --force portainer_portainer

docker pull portainer/agent:latest

docker service update --image portainer/agent:latest --force portainer_agent