Longhorn is a stable open source Kubernetes distributed block storage that can be used in production. Configuring longhorn is not very complicated, but there are a few caveats to consider when using it with MicroK8S on an Ubuntu 20.04 box
This guide assumes you have set up your kubernetes cluster on a single node as per our previous guide.
After that, we can proceed as follows to set up Longhorn
1.Install nfs-common and open-iscsi
root@vmi663745:~# apt install nfs-common open-iscsi -y
2.Start and Enable nfs-common
root@vmi663745:~# systemctl start nfs-common
root@vmi663745:~# systemctl enable nfs-common
In case you encounter an error that nfs-common is masked, please check the Troubleshoot section of this guide.
2. Start and Enable open-iscsi
root@vmi663745:~# systemctl start open-iscsi; systemctl enable open-iscsi
NOTE: For open-iscsi service you may first need to create an iscsi target – client setup for the service to start. If you start the service, and it fails with no errors, please check our guide on how to make it work. Ideally, if you restart open-iscsi and when checking status you get similar output as below, you need to check the guide to successfully start this service
root@vmi663745:~# systemctl status open-iscsi
● open-iscsi.service - Login to default iSCSI targets
Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2021-10-11 16:08:13 CEST; 2min 20s ago
├─ ConditionDirectoryNotEmpty=|/etc/iscsi/nodes was not met
└─ ConditionDirectoryNotEmpty=|/sys/class/iscsi_session was not met
3. Install Longhorn
There are 3 ways to install longhorn as per Longhorn Docs. In our case, we will install it via Helm Chart
- Add the Longhorn Helm repository:
root@vmi663745:~# helm repo add longhorn https://charts.longhorn.io
- Update Helm repo
root@vmi663745:~# helm repo update
- Create a namespace and install Longhorn
root@vmi663745:~# kubectl create namespace longhorn-system
root@vmi663745:~# helm install longhorn longhorn/longhorn --namespace longhorn-system
Ideally, Longhorn should start all pods but often times – for reasons I have not yet know – it fails to successfully deploy longhorn-driver and thus several pods can’t work yet. Check the Troubleshooting issue no. 3 below for a fix
4. Set Longhorn as the default storage class
root@vmi663745:~# kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Since I will not be using the microk8s-hostpath storage provider, I can proceed to remove it
root@vmi663745:~# microk8s.disable storage
Troubleshoot
1.nfs-common service is masked
Error:
Failed to start nfs-common.service: Unit nfs-common.service is masked.
Symptoms:
Error occurs when you run systemctl start nfs-common for the first time after installation
Solution:
- Check if the nfs-common.service file exists as below and points to /dev/null
root@vmi663745:~# ls -l /lib/systemd/system/nfs-common.service
lrwxrwxrwx 1 root root 9 May 24 22:51 /lib/systemd/system/nfs-common.service -> /dev/null
- Remove the file
root@vmi663745:~# rm -f /lib/systemd/system/nfs-common.service
- Reload daemons then restart nfs-common
root@vmi663745:~# systemctl daemon-reload
root@vmi663745:~# systemctl start nfs-common
root@vmi663745:~# systemctl status nfs-common
● nfs-common.service - LSB: NFS support files common to client and server
Loaded: loaded (/etc/init.d/nfs-common; generated)
Active: active (running) since Mon 2021-10-11 15:23:16 CEST; 15s ago
Docs: man:systemd-sysv-generator(8)
Process: 27904 ExecStart=/etc/init.d/nfs-common start (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 9485)
Memory: 55.9M
CGroup: /system.slice/nfs-common.service
├─27926 /sbin/rpc.statd
└─27941 /usr/sbin/rpc.idmapd
#########- Enable nfs-common
root@vmi663745:~# systemctl enable nfs-common
2.MicroK8S cluster does not install packages from Helm
Error:
INSTALLATION FAILED: Kubernetes cluster unreachable: Get “http://localhost:8080/version?timeout=32s”: dial tcp 127.0.0.1:8080: connect: connection refused
Solution:
Set the kubeconfig file
root@vmi663745:~# microk8s.kubectl config view --raw > ~/.kube/config
3. Longhorn driver does not start
Issue:
After successfully installing Longhorn via kubectl or Helm method, the longhorn-driver pod starts but keeps crashing
Error logged under pod description is as below
time=”2021-09-23T13:15:38Z” level=error msg=”failed to get arg root-dir. Need to specify \”–kubelet-root-dir\” in your Longhorn deployment yaml.: failed to get kubelet root dir, no related proc for root-dir detection, error out”
time=”2021-09-23T13:15:38Z” level=fatal msg=”Error deploying driver: failed to start CSI driver: failed to get arg root-dir. Need to specify \”–kubelet-root-dir\” in your Longhorn deployment yaml.: failed to get kubelet root dir, no related proc for root-dir detection, error out”
Solution:
A workaround I found in Github Issues is running the command below
root@vmi663745:~# curl https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml | sed -e 's/#- name: KUBELET_ROOT_DIR/- name: KUBELET_ROOT_DIR/g' -e 's$# value: /var/lib/rancher/k3s/agent/kubelet$ value: /var/lib/kubelet$g' | kubectl apply -f -
After that, longhorn-driver runs and other related pods also are created and run successfully
