Configuring Kubernetes Event Exporter for OpenSearch

A comprehensive guide to seamlessly push logs from Kubernetes to OpenSearch

Key Insights

Deployment and Configuration: Deploying the exporter via Helm or directly using YAML ensures Kubernetes events are captured.
OpenSearch Integration: Configuring endpoints, authentication, and TLS settings plays a critical role in secure and efficient log transfer.
Verification and Troubleshooting: Monitoring pods, logs, and OpenSearch indexing is vital to ensure proper operation.

Introduction

Pushing logs from Kubernetes to OpenSearch using the Kubernetes Event Exporter allows you to monitor, search, and analyze cluster events in real time. This comprehensive guide outlines deployment, configuration, and verification steps to ensure that you can successfully forward logs from Kubernetes events into your OpenSearch environment.

Deployment Methods

Using Helm for Deployment

Helm simplifies the deployment process by allowing you to utilize pre-configured charts. Start by adding the necessary Helm repository and then deploying the exporter.

Step-by-Step Guide with Helm

Add the Helm Repository:


# Add the repository hosting the Kubernetes Event Exporter chart
helm repo add resmo https://chart.resmo.io
helm repo update

Create a Values File:

Create a values.yaml file that includes the configuration settings for OpenSearch. For example:


# Sample values.yaml
receivers:
  - name: "opensearch"
    opensearch:
      url: "https://opensearch-cluster-master:9200"
      index: "eks-kubernetes-events"
      username: "admin"
      password: "admin"
      tls:
        insecure: true  # Change to false when using valid TLS certs

routes:
  - match:
      - receiver: "opensearch"

Install the Exporter:


# Deploy the exporter in the kube-system or a dedicated namespace
helm install kubernetes-event-exporter resmo/kubernetes-event-exporter \
  --namespace kube-system \
  --values values.yaml

Update an Existing Deployment:

If the exporter is already deployed, use the following command to upgrade your configuration:


helm upgrade kubernetes-event-exporter resmo/kubernetes-event-exporter \
  --namespace kube-system \
  --values values.yaml

Direct Deployment with YAML

For those who prefer not to use Helm, deploying the Kubernetes Event Exporter via YAML is a straightforward alternative. Prepare your Deployment manifest along with a ConfigMap that supplies the necessary configuration settings.

YAML Deployment Example

Begin with a Deployment configuration specifying your container image, arguments, and volume mounts for the configuration file:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: event-exporter
  template:
    metadata:
      labels:
        app: event-exporter
    spec:
      serviceAccountName: event-exporter
      containers:
      - name: event-exporter
        image: opsgenie/kubernetes-event-exporter:0.7
        imagePullPolicy: IfNotPresent
        args:
        - -conf=/data/config.yaml
        volumeMounts:
        - mountPath: /data
          name: cfg
  volumes:
  - name: cfg
    configMap:
      name: event-exporter-cfg

Next, create a ConfigMap containing the OpenSearch settings:


apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    receivers:
      - name: "opensearch"
        opensearch:
          url: "https://opensearch-cluster-master:9200"
          index: "eks-kubernetes-events"
          username: "admin"
          password: "admin"
          tls:
            insecure: true
    routes:
      - match:
          - receiver: "opensearch"

Finally, apply these configurations:


kubectl apply -f deployment.yaml
kubectl apply -f configmap.yaml

OpenSearch Configuration

Integration with OpenSearch

The successful integration of Kubernetes events with OpenSearch involves ensuring that OpenSearch is properly set up to receive logs:

OpenSearch Setup

Endpoint and Index: Verify that the OpenSearch endpoint specified in your configuration points to a reachable OpenSearch cluster, and set the appropriate index (e.g., eks-kubernetes-events) where logs will be stored.
Authentication: Configure credentials (username and password) correctly. Use the options provided in your configuration file to manage these securely.
TLS Configuration: When using secure connections, ensure that TLS settings are correctly defined (e.g., insecure: true for testing, otherwise provide valid certificates).
Index Patterns: In OpenSearch, create an index pattern such as logs-collectord-* with @timestamp as the time field to facilitate proper log visualization.

Verification and Troubleshooting

Steps to Verify Exporter Functionality

After deployment and configuration, it is important to verify that your setup is functioning as expected:

Verify Kubernetes Deployment

Check Pod Status: Ensure that the exporter pod is running:


kubectl get pods -n kube-system -l app=kubernetes-event-exporter

Inspect Logs: Check the logs from the event exporter pod for any errors or confirmation of successful connection:
```
kubectl logs -f pod/[exporter-pod-name] -n kube-system
      
```

Test Event Generation

Simulate Events: Generate events by performing actions such as creating/deleting pods.
Review OpenSearch Dashboards: Log in to your OpenSearch dashboard and look for the events in the target index (eks-kubernetes-events).

Common Troubleshooting Tips

Version Compatibility: Ensure that the Kubernetes Event Exporter version is compatible with your OpenSearch libraries. Mismatches can lead to EOF errors or log drop issues.
Client-Side Throttling: Adjust settings if encountering client-side throttling warnings. Modify event cut off ages if events are dropped.
Configuration Verification: Revisit all configuration files (values files, deployment YAMLs, and ConfigMaps) to ensure that the endpoints, credentials, and TLS settings are correct.

Comprehensive Configuration Table

Component	Configuration	Command/Notes
Kubernetes Event Exporter	Deployment via Helm or YAML ConfigMap with OpenSearch settings	`helm install kubernetes-event-exporter --values values.yaml`
Helm Values File	OpenSearch endpoint (url) Index: "eks-kubernetes-events" Authentication: username & password TLS settings	Modify `values.yaml` accordingly.
Deployment Manifest	Container image: opsgenie/kubernetes-event-exporter Arguments pointing to config.yaml Volume mounting for configuration	`kubectl apply -f deployment.yaml`
Verification	Pod status check Logs inspection OpenSearch dashboard review	`kubectl get pods \| kubectl logs`