How to use the Humio Operator to run Humio on Kubernetes

This blog was originally published Nov. 12, 2020 on humio.com. Humio is a CrowdStrike Company.

Running distributed stateful systems on Kubernetes can be a challenge — but it doesn’t have to be. The Humio Operator facilitates the configuration and deployment, and eases the management of your Humio clusters running on Kubernetes. In our previous blog post, we introduced the Humio Operator. In this second of a two-part series, we walk you through how to use the Humio Operator to run Humio on Kubernetes.

To run Humio on Kubernetes, you need Apache ZooKeeper and Apache Kafka. For full end-to-end instructions on preparing and installing Humio on Kubernetes, see the Operator Quick Start Guide. For simplicity’s sake, we will create a Humio Cluster using Persistent Volumes, we assume we have Zookeeper and Kafka, and we will use ‘port-forward’ to connect to our cluster.

Install CRDs

Before the Humio Operator can be installed, we must create our Custom Resource Definitions. The CRDs define the resources that the Humio Operator will manage, which include HumioClusters, HumioExternalClusters, HumioIngestTokens, HumioParsers, and HumioRepositories.

Find the version of the Humio Operator that we want to install, and create the CRDs:

export HUMIO_OPERATOR_VERSION=0.1.1
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioexternalclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioingesttokens.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioparsers.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humiorepositories.yaml

Install the Humio Operator

The Humio Operator can be installed with Helm.

First add the helm repo:

helm repo add humio-operator https://humio.github.io/humio-operator

And install:

helm install humio-operator humio-operator/humio-operator --version="${HUMIO_OPERATOR_VERSION}"

A full list of Helm values can be found in the installation documentation.

Create a Humio Cluster

A HumioCluster resource can now be created. The HumioCluster will start up our Humio pods. We need to point this cluster to our existing ZooKeeper and Kafka clusters.

cat <apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
name: example-humiocluster
spec:
autoRebalancePartitions: true
tls:
enabled: false
dataVolumePersistentVolumeClaimSpecTemplate:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 10Gi
environmentVariables:
- name: "SINGLE_USER_PASSWORD"
value: "develop3r"
- name: "HUMIO_JVM_ARGS"
value: "-Xss2m -Xms256m -Xmx1536m -server -XX:+UseParallelOldGC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Dzookeeper.client.secure=false"
- name: "ZOOKEEPER_URL"
value: ""
- name: "KAFKA_SERVERS"
value: ""
EOF
humiocluster.core.humio.com/example-humiocluster created

Once the HumioCluster is created, it will go into the state of Bootstrapping until all Humio pods are up:

kubectl get humiocluster
NAME                   STATE           NODES   VERSION
example-humiocluster   Bootstrapping

Once all Humio nodes are up, the Operator will have access to the Humio cluster and update the status of the HumioCluster custom resource including node count and version. At this point it will automatically balance our digest and storage partitions and be ready to create other Humio resources.

kubectl get humiocluster
NAME                   STATE     NODES   VERSION
example-humiocluster   Running   3       1.16.0--build-293562267--sha-39bbd8579eba9d6527e8b1579cdb9504a426a411

We can access the Humio UI by using kubectl port-forward. For production installations, we would rely on Ingress.

kubectl port-forward svc/example-humiocluster 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080

Now go to http://localhost:8080 and login using the username “developer” and password “develop3r”, which we set during the cluster creation. When we go to the Cluster Administration under the settings button in the top right corner, we see the list of our pods and that the digest and storage partitions have been balanced:

Create a Repository, Parser, and Ingest Token

Now that we have a running cluster, we can create some other resources that are managed by the Operator. Let’s start with a Repository.

cat <

cat <[^]]+)]/
        | /^(?<@timestamp>.*)[warn].*/
        | parseTimestamp(format="yyyy/MM/dd' 'HH:mm:ss", [email protected]);
      parseJson();
      * | kvParse()
    }
EOF
humioparser.core.humio.com/fluentbit-parser created

cat <

export INGEST_TOKEN=$(kubectl get secret fluentbit-ingest-token -o template --template '{{.data.token}}' | base64 -D)

helm repo add humio https://humio.github.io/humio-helm-charts
helm repo update

cat >> humio-agent.yaml<

helm install humio humio/humio-helm-charts --set humio-fluentbit.token=$INGEST_TOKEN -f humio-agent.yaml