Skip to main content

Troubleshooting

General Troubleshooting guide

Kubectl / helm commands fail​

If you receive an error as shown below when trying to invoke kubectl or helm commands, it is most likely caused by the KUBECONFIG variable not being set.

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

The installation script attempts to create $HOME/.kube/config and sets this in the $HOME/.bashrc

You may need to log out and log back in, to create a new shell terminal session which will contain the correct KUBECONFIG variable, or run source ~/.bashrc to reload you shell config, or manually set it with export KUBECONFIG=~/.kube/config

NOTE: It is not advisable to run kubectl and helm commands as root (or sudo kubectl ... or sudo helm ....

The user-provided configs are sufficient for interacting with the k3s cluster and its resources.

Make sure that you $KUBECONFIG is set to a file that is accessible by your system user with the correct permissions.

You can view the functions responsible for setting kubeconfig here

$KUBECONFIG file has expired​

With the K3s installation, the kubeconfig can "expire" causing issues authenticating with the kubeapi.

eg response would be like this:

kubectl get secret chronicle-feed-eth-keys -n chronicle-feed -o jsonpath="{.data.ethFrom}" | base64 --decode
E1002 15:13:49.343237 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.345241 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.349349 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.351739 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.353627 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

This happens because we are interacting with the k3s cluster as a non-root user (best practice), and we are getting our kubeconfig file rancher at /etc/rancher/k3s/k3s.yaml. The client-certificate-data gets refreshed from time to time, in which case you will need to create a new kubeconfig file (normally once a year).

To fix this, run the following commands:

# as your validator user (eg demo-user)

rm $KUBECONFIG
sudo systemctl restart k3s
sudo cp /etc/rancher/k3s/k3s.yaml $KUBECONFIG
sudo chown $USER $KUBECONFIG

You should be able to authenticate with the cluster and proceed with kubectl and helm commands.

Validator restarts in a loop on chart 0.8.2​

Since app 0.81.0 the validator has no built-in fallback configuration. If at start it cannot read the on-chain config registry through your Ethereum RPC, or cannot download its configuration from IPFS, the container exits with code 0 and Kubernetes restarts it. The pod shows Completed and then CrashLoopBackOff instead of Error, and the log ends with could not run app: no working config found. This applies to new installs as well as upgrades.

  • Read the previous run: kubectl logs deployment/ghost -n $FEED_NAME --previous and kubectl logs deployment/ghost-vao -n $FEED_NAME --previous.
  • Check that ghost.rpcUrl is an Ethereum mainnet RPC that answers from the node (the config registry is on Ethereum mainnet), and that outbound HTTPS (port 443) is open.
  • If the log shows ConfigDoesNotExist, the config registry has no configuration for your validator address yet. The Chronicle team adds it when it onboards your address, and app 0.81.0 cannot start before that. Contact the Chronicle team.
  • If the log shows file name too long, app 0.81.0 tried to read your key passphrase as a file name and the passphrase is too long for one (over 255 bytes once URL encoded, where a space or a non-ASCII character takes 3 bytes or more). That log line contains the passphrase itself, so delete it from any log you share, including the debug bundle below. Roll back and contact the Chronicle team.
  • To go back to the chart you ran before while you investigate, run helm history $FEED_NAME -n $FEED_NAME, pick the last revision whose CHART column is older than validator-0.8.2, and run helm rollback $FEED_NAME <revision> -n $FEED_NAME. Without a revision number, helm rollback returns to the revision just before the current one, which is still chart 0.8.2 if you ran the upgrade more than once.

Debug Bundle​

If you need further debugging, please retrieve the container logs, and some base info and provide it to the chronicle team for assistance:

tip

Make sure that you set FEED_NAME to match your feed in question, and run these commands from the correct user so that $HOME/$FEED_NAME/generated-values.yaml resolves to the correct file.

export FEED_NAME=<CHANGE_ME>
cd /tmp
kubectl logs deployment/ghost -n $FEED_NAME > ghost.log
kubectl logs deployment/ghost-vao -n $FEED_NAME > ghost-vao.log
kubectl get pods,deployment,service,secrets --all-namespaces > kbom.txt
helm list --all-namespaces > all-helm-releases.log
tar czf chronicle-debug.tar.gz ghost.log ghost-vao.log all-helm-releases.log kbom.txt $HOME/$FEED_NAME/generated-values.yaml

The above commands will get logs for each of the pods running, the state of the services, as well as the generated-values.yaml for your feed, and bundle it into a tarball chronicle-debug.tar.gz (Please change the values / paths accordingly)