Customising kubectl Output to Cleanly Get the Status of a Node

Background

I am working to build an experimental Kubernetes cluster with Ansible.

Some steps in the Ansible configuration are asynchronous, so I need to use do..until tasks to delay further processing of the configuration until the asynchronous steps are completed.

do..until waits on a condition to become true before considering the task to be complete and applying the remaining tasks. This could, for example, be waiting on provisioned nodes to be in a Ready state.

I looked at a number of examples that used kubectl get nodes ansible_facts['hostname'] and piped the output through awk and/or grep. Others used Jinja2 filters to filter the standard output (stdout). I felt there had to be a better way…

Customising the output of kubectl

The default output of kubectl get nodes look like:

$ kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes k8scp01
NAME     STATUS   ROLES                  AGE   VERSION
k8scp01  Ready    control-plane,master   62m   v1.22.1

Remove Column Headings

The first step is eliminating the column headings. We can do this simply by adding --no-headers to the kubectl command:

$ kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes k8scp01 --no-headers

k8scp01  Ready   control-plane,master   64m   v1.22.1

Inspect the JSON API data

I thought the second step would be returning the single value needed. I thought it would be possible to return just the STATUS column. Before we can reach an elegant solution, we have to grapple with some JSON.

I noted that kubectl accepts a parameter called --output and various values can be appended to that. These include, for example, json to return raw JSON API output.

Here is an excerpt of what gets returned when we append --output=json to the kubectl command:

{
	"apiVersion": "v1",
	"kind": "Node",
	
	<snip>

	"status": {

		<snip>
		
		"conditions": [
			{
				"lastHeartbeatTime": "2021-08-23T18:00:32Z",
				"lastTransitionTime": "2021-08-23T18:00:32Z",
				"message": "Calico is running on this node",
				"reason": "CalicoIsUp",
				"status": "False",
				"type": "NetworkUnavailable"
			},
			{
				"lastHeartbeatTime": "2021-08-23T18:01:34Z",
				"lastTransitionTime": "2021-08-23T17:39:18Z",
				"message": "kubelet has sufficient memory available",
				"reason": "KubeletHasSufficientMemory",
				"status": "False",
				"type": "MemoryPressure"
			},
			{
				"lastHeartbeatTime": "2021-08-23T18:01:34Z",
				"lastTransitionTime": "2021-08-23T17:39:18Z",
				"message": "kubelet has no disk pressure",
				"reason": "KubeletHasNoDiskPressure",
				"status": "False",
				"type": "DiskPressure"
			},
			{
				"lastHeartbeatTime": "2021-08-23T18:01:34Z",
				"lastTransitionTime": "2021-08-23T17:39:18Z",
				"message": "kubelet has sufficient PID available",
				"reason": "KubeletHasSufficientPID",
				"status": "False",
				"type": "PIDPressure"
			},
			{
				"lastHeartbeatTime": "2021-08-23T18:01:34Z",
				"lastTransitionTime": "2021-08-23T18:00:34Z",
				"message": "kubelet is posting ready status. AppArmor enabled",
				"reason": "KubeletReady",
				"status": "True",
				"type": "Ready"
			}
		],
		
		<snip>
		
	}
}

Somewhere here is the status of our node.

The JSON API returns multiple instances of status.condition, each with a type and a boolean status value.

By observation, I see that when our node reaches a condition of “STATUS: Ready” by the conventional kubectl get nodes output, we see a status.condition with a type of Ready and a value of True.

That is good enough for our purposes!

Returning a “custom-column”

The kubectl --output parameter also accepts a parameter called custom-columns which can be specified with a JSONPath. That means we should be able to get a succinct answer with something like $.status.conditions[?(@.type=='Ready')].status:

$ kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes --no-headers -output=custom-columns=STATUS:status.conditions[?(@.type==\'Ready\')].status
True

That gives us a nice easily comparable value for use with Ansible.

Ansible Example

- name: wait until node is in ready state
  environment:
    KUBECONFIG: /etc/kubernetes/admin.conf
  shell: |
    kubectl get nodes {{ ansible_hostname }} --no-headers -o custom-columns=STATUS:status.conditions\[\?\(\@.type==\"Ready\"\)\].status
  register: node_status
  until: node_status.stdout == 'True'
  delay: 10
  retries: 30

In this example, we will wait up to five minutes for the node to get a status type of Ready with a value of True.

Conclusion

This is one small building block in creating a Kubernetes cluster with Ansible. In future I hope to add full example on GitLab with a full write-up here.