Sensu - Getting Started

Monitoring is often a sore topic. It’s crucial to get it right and get a grasp of what’s going on with your infrastructure. The more details you have during a failure the quicker you’ll act and recover from it. There are many solutions out there like Nagios, Shinken, Zabbix, Icinga to just name a few. And they are all great. Today I’ll show you Sensu which is one of the options I really like. It was rewritten in Go and has features that in my opinion makes it stand out:

  • Simple - two native binaries talking. One is an agent (host being monitored) and the other a backend (Sensu’s “mothership”).
  • Powerful cli called sensuctl.
  • Feels like Kubernetes with its declarative configuration.
  • Packs all the right stuff you need in a monitoring platform - from the check to the alert with great options to automate and remediate in between.
  • Event driven - it’s so much more than just alerting about abnormal condition since every event can be handled in any way you like.
  • Feels like a framework, not just a tool.
  • Easy to extend via its powerful asset management.

And those are just a few of its great treats. To get started I would suggest to walk through a simple docker setup with one backend and an agent. According to its documentation at https://sensu.io/ that would involve a few steps:

Start Sensu Backend

$ sudo docker network create sensu
$ sudo docker volume create sensu-backend-data
$ sudo docker run -d --rm --name sensu-backend \
  --network sensu -p 8080:8080 -p 3000:3000 \
  -v sensu-backend-data:/var/lib/sensu \
  sensu/sensu:6.2.5 sensu-backend start
11bbb27c3
$ sudo docker run -d --rm --network sensu -p :3030 \
  sensu/sensu:6.2.5 sensu-agent start \
  --backend-url ws://sensu-backend:8081 --deregister \
  --keepalive-interval=5 --keepalive-warning-timeout=10 --subscriptions linux
$ curl -s http://localhost:8080/version
{"etcd":{"etcdserver":"3.3.13","etcdcluster":"3.3.0"},"sensu_backend":"6.2.5"}
$ _

Install Sensu cli and connect to the backend

$ curl -LO https://s3-us-west-2.amazonaws.com/sensu.io/sensu-go/6.2.5/sensu-go_6.2.5_darwin_amd64.tar.gz
$ sudo tar -xzf sensu-go_6.2.5_darwin_amd64.tar.gz -C /usr/local/bin/
$ sensuctl configure -n --url http://127.0.0.1:8080 \
  --username admin \
  --password 'P@ssw0rd!' \
  --namespace default
$ sensuctl cluster health
ID                NAME     ERROR   HEALTHY
8927110dc66458af  default          true
$ sensuctl cluster id
sensu cluster id: "227b26eb-26c4-46b2-bc7c-8c080b072e6b"
$ _

Connect an Agent and configure one simple check

$ sensuctl asset add sensu/monitoring-plugins:2.2.0-1
fetching bonsai asset: sensu/monitoring-plugins:2.2.0-1
added asset: sensu/monitoring-plugins:2.2.0-1
$ sensuctl check create ntp \
  --runtime-assets "sensu/monitoring-plugins" \
  --command "check_ntp_time -H time.nist.gov --warn 0.5 --critical 1.0" \
  --output-metric-format nagios_perfdata \
  --publish="true" --interval 30 --timeout 10 --subscriptions linux
$ sensuctl event list
ENTITY        CHECK      OUTPUT                                                                    STATUS  SILENCED   TIMESTAMP
a749e3a10d86  keepalive  Keepalive last sent from a749e3a10d86 at 2019-09-11 15:34:25 +0000 UTC    0       false      2019-09-11 08:34:25 -0700 PDT
a749e3a10d86  ntp        NTP OK: Offset -0.03375908732 secs|offset=-0.033759s;0.500000;1.000000;   0       false      2019-09-11 08:34:22 -0700 PDT
$ _

When just starting it’s nice to follow those steps and make the UI and cli work quickly to get a sense of accomplishment. Sensu does make this process easy as you can see in those few steps. Take your time and click through the UI which will be available on port :3000. Also get a feel for its cli interface:

➜  ~ sensuctl
sensuctl controls Sensu instances

Usage:	sensuctl COMMAND

Flags:
      --api-url string             host URL of Sensu installation
      --cache-dir string           path to directory containing cache & temporary files (default "/Users/kstaykov/Library/Caches/sensu/sensuctl")
      --config-dir string          path to directory containing configuration files (default "/Users/kstaykov/.config/sensu/sensuctl")
  -h, --help                       help for sensuctl
      --insecure-skip-tls-verify   skip TLS certificate verification (not recommended!)
      --namespace string           namespace in which we perform actions (default "default")
      --timeout duration           timeout when communicating with sensu backend (default 15s)
      --trusted-ca-file string     TLS CA certificate bundle in PEM format

Commands:
  completion           Output shell completion code for the specified shell (bash or zsh)
  configure            Initialize sensuctl configuration
  create               Create or replace resources from file or URL (path, file://, http[s]://), or STDIN otherwise.
  delete               Delete resources from file or STDIN
  describe-type        Print details about the supported API resources types
  dump                 Dump resource definitions to JSON or YAML
  edit                 Edit resources interactively
  env                  Display the commands to set up the environment used by sensuctl
  logout               Logout from sensuctl
  prune                Deletes resources that do not appear in the configs from file or URL (path, file://, http[s]://), or STDIN otherwise.
  version              Show the sensuctl version information

Management Commands:
  api-key              Manage apikeys
  asset                Manage assets
  auth                 Manage authentication drivers
  check                Manage checks
  cluster              Manage sensu cluster
  cluster-role         Manage cluster roles
  cluster-role-binding Manage cluster role bindings
  command              Manage sensuctl commands
  config               Modify sensuctl configuration
  entity               Manage entities
  event                Manage events
  filter               Manage filters
  handler              Manage handlers
  hook                 Manage hooks
  license              Manage enterprise license
  login                Authenticate sensuctl to Sensu using the provided arguments
  mutator              Manage mutators
  namespace            Manage namespaces
  role                 Manage roles
  role-binding         Manage role bindings
  secret               Manage secrets
  silenced             Manage silenced subscriptions and checks
  tessen               Manage tessen configuration
  user                 Manage users

Run 'sensuctl COMMAND --help' for more information on a command.
➜  ~

I won’t get into more details in this getting started article but do you remember when I said that it feels like Kubernetes? I’ll give you a small hint. Check out the sensuctl dump all command. This is how anything you configure can be dumped to declarative code that you can put in version control. You can also use sensuctl create -f <yaml file> to bring that configuration into play on the backend you have running. Don’t forget to sensuctl login first.

Categories:

Updated: