Chaos Engineering on IBM Cloud

Objective

  • Install Gremlin on your IBM Cloud container platforms.
  • Experiment with chaos engineering

Services Used

  • IBM Cloud Kubernetes Service or Red Hat OpenShift on IBM Cloud
  • Gremlin

Before you begin

This tutorial assumes you have an existing IBM Cloud Kubernetes Services cluster or Red Hat OpenShift on IBM Cloud cluster. If you don’t have an existing environment, then use the following tutorials to build one:

  • For clusters on IBM Cloud Kubernetes Service, please visit here.
  • For clusters on Red Hat OpenShift on IBM Cloud, please visit here.
Cat, testing your cluster

Create an account with Grelim

Step 1 — Register with Gremlin

  1. Go to Gremlin and select your service.
  2. Login to your account.

Step 2 — Download the Gremlin certificates

  1. 1. Return to the Gremlin dashboard and copy team id, ­­­­­ click Account > Team Settings > Configuration > Team ID > Copy ID
  2. Create an environment variable in your desktop’s shell to store your Gremlin store id
    export GREMLIN_TEAM_ID=”<gremlin_team_ID>”

Step 3 — Download the Gremlin certificates to your desktop

  1. In the Gremlin dashboard, click Account> Team Settings > Configuration > Certificates > Download
    Note: You will need team manager access role.
  2. Locate, store, and rename the certificate files to as following somewhere accessible by your desktop’s shell:
    i. <team-name>.priv_key.pem to gremlin.key
    ii. <team-name>.pub_cert.pem to gremlin.cert

Option A — Deployment steps for IBM Cloud Kubernetes Service

Deploying Gremlin on IBM Cloud Kubernetes Service cluster

Gremlin must be deployed on each cluster you wish to attack. In order for your clusters and containers to be targetable, the Gremlin agent must be registered with the Gremlin Control Plane.

  1. From your desktop’s shell, connect to your IBM Cloud Kubernetes Service instance.
  2. Create a namespace for Gremlin.
    kubectl create namespace gremlin
  3. Create a Kubernetes secret for Gremlin
kubectl -n gremlin create secret generic gremlin-team-cert --from-file=/<fullpath>/gremlin.cert --from-file=/<fullpath>/gremlin.key
export GREMLIN_CLUSTER_ID=”<your IBM Cloud Kubernetes Service cluster ID>”
helm install gremlin gremlin-beta/gremlin \
--namespace gremlin \
--set runtime.name=containerd \
--set gremlin.hostPID=true \
--set gremlin.usePodSecurityPolicy=false \
--set gremlin.secret.managed=true \
--set gremlin.secret.teamID=$GREMLIN_TEAM_ID \
--set gremlin.secret.clusterID==$GREMLIN_CLUSTER_ID \
--set-file gremlin.secret.certificate=gremlin.cert \
--set-file gremlin.secret.key=gremlin.key \
--set gremlin.apparmor=unconfined

Option B — Deployment steps for Red Hat OpenShift on IBM Cloud

Deploying Gremlin on your Red Hat OpenShift on IBM Cloud cluster on IBM Cloud

NOTE: This guide assumes that you will be installing Gremlin into its own namespace. You can start a new project and namespace with the following command. All subsequent oc create commands in this guide leave out the — namespace argument, assuming that you wish to install Gremlin in the current OpenShift project.

  1. From your desktop’s shell, connect to your Red Hat OpenShift on IBM Cloud instance.
  2. Create a namespace for Gremlin.
    oc new-project gremlin
  3. From your Red Hat OpenShift on IBM Cloud cluster console get your cluster ID and export it as an environment variable in your desktop’s shell.
    export GREMLIN_CLUSTER_ID=”<your Red Hat OpenShift on IBM Cloud cluster ID>”
  4. Create a Kubernetes secret containing the key-pair and your team ID
oc create secret generic gremlin-secret \
--from-literal=GREMLIN_TEAM_ID=$GREMLIN_TEAM_ID \
--from-literal=GREMLIN_CLUSTER_ID=$GREMLIN_CLUSTER_ID \
--from-file=gremlin.cert=$PATH_TO_CERTIFICATE \
--from-file=gremlin.key=$PATH_TO_PRIVATE_KEY
oc create serviceaccount gremlin -n gremlin
oc adm policy add-scc-to-user privileged -z gremlin -n gremlin
oc create -f gremlin-scc.yamloc adm policy add-scc-to-user gremlin -z gremlin
oc create -f gremlin-daemonset.yaml
oc create -f chao-service-account.yaml
oc create -f chao-deployment.yaml

Setting up a Gremlin attack

  1. Login to the Gremlin’s Dashboard
  2. In the left menu, click on Attacks to define an attack
Click on Attacks to define an attack
What type of target do you want to attack?
Choose the containers to target
Choose percent of targets to impact
Choose a CPU type Gremlin
Set the impact of the attack

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Robert Barron

Robert Barron

Lessons from the Lunar Landing, Shuttle to SRE | AIOps, ChatOps, DevOps and other Ops | IBMer, opinions are my own