7.6. Troubleshooting K3s

This section offers troubleshooting information when the K3s installer or the K3s configuration checker fail.

7.6.1. Troubleshooting the K3s Installer

The LOCKSS Installer's install-k3s script installs K3s by executing Rancher's official K3s installer after making sure many firewall and DNS issues are resolved 1. However, the installer can still run into issues and fail. Some of the error messages you might encounter are documented below, but you may need to refer to the official K3s documentation or use a search engine to look up the specific error message.

7.6.1.1. Failed to apply container_runtime_exec_t

In some Fedora systems, you may see an error message similar to the following:

[ERROR]  Failed to apply container_runtime_exec_t to /usr/local/bin/k3s, please install:
    yum install -y container-selinux selinux-policy-base
    yum install -y https://rpm.rancher.io/k3s/stable/common/centos/7/noarch/k3s-selinux-0.2-1.el7_8.noarch.rpm

The specific commands and version numbers may vary from the example above.

To resolve this problem:

  1. Run the recommended commands as root 6.

  2. Re-run install-k3s 1.

7.6.1.2. k3s-selinux requires container-selinux

In some Oracle Linux 7 systems, you may see an error message similar to the following:

Error: Package: k3s-selinux-0.3-0.el7.noarch (rancher-k3s-common-stable)
           Requires: container-selinux >= 2.107-3
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

The specific commands and version numbers may vary from the example above.

This can occur in environments where the Oracle Linux 7 Addons Yum repository is not enabled by default, so Rancher's official K3s installer is unable to install the package container-selinux automatically.

To resolve this problem:

  1. Run the following command as root 6:

    yum-config-manager --enable ol7_addons
    
  2. Re-run install-k3s 1.

7.6.2. Troubleshooting the K3s Configuration Checker

After installing K3s with install-k3s 1 and successfully running check-k3s 2, you can run the following command as root 6:

k3s check-config

This configuration checker runs through a more extensive series of tests, covering "required", "generally necessary", and "optional" aspects for K3s to operate.

As a rule of thumb, if k3s check-config ends successfully with STATUS: pass, there is a good chance the K3s cluster is configured correctly.

Some failures, especially in "optional" aspects, may not actually prevent the cluster from working normally in the limited ways the LOCKSS system uses Kubernetes, but if possible they should be addressed. Some of the error messages you might encounter are documented below, but you may need to refer to the official K3s documentation or use a search engine to look up the specific error message.

7.6.2.1. iptables should be older than v1.8.0 or in legacy mode

In some instances, you may encounter an error message similar to the following:

iptables v1.8.4 (nf_tables): should be older than v1.8.0 or in legacy mode (fail)

This error message is generally spurious, because the LOCKSS Installer should have previously detected and offered to correct this issue in the circumstances where it applies, and Rancher has a documented bug report that the K3s configuration checker keeps reporting this issue even in circumstances where it does not apply 4.

  • If check-k3s ran successfully 2, your K3s cluster is probably running normally and you can ignore this error message even if you receive it.

  • If your system is running iptables version 1.8.0 or later in nf_tables mode via Alternatives, as can be the case in some Debian or Ubuntu systems, iptables needs to be switched to legacy mode via Alternatives. The configure-firewall script called by install-k3s is supposed to detect this condition and offer to fix it for you 1. See Troubleshooting iptables.

7.6.2.2. User namespaces disabled

In the RHEL/CentOS family of operating systems, you may receive the following error message:

RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line

To resolve this issue 5:

  1. Edit the file /etc/default/grub as root 6.

    1. Look for the line beginning with GRUB_CMDLINE_LINUX=, for example:

      GRUB_CMDLINE_LINUX="no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 elevator=noop crashkernel=auto"
      
    2. Add user_namespace.enable=1 to the space-separated list of boot arguments, for instance:

      GRUB_CMDLINE_LINUX="user_namespace.enable=1 no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 elevator=noop crashkernel=auto"
      
  2. Run the following command as root:

    grub2-mkconfig -o /boot/grub2/grub.cfg
    
  3. Reboot the system.

  4. Re-run k3s check-config 3.

7.6.2.3. swap should be disabled

This warning can be ignored:

swap: should be disabled

7.6.2.4. CONFIG_INET_XFRM_MODE_TRANSPORT missing

This warning can be ignored:

CONFIG_INET_XFRM_MODE_TRANSPORT: missing

Footnotes

1

See install-k3s.

2

See check-k3s.

3

See k3s-check-config.

4

References:

5

References:

6

See Running Commands as root.