Mike Lanyon's Blog

Creative & Technology: A Partnership - DevOpsDays MSN 2016

Wed, 02 Nov 2016 00:00:00 +0000

"The premise that dev and ops are at odds is a fallacy."

This talk has been brewing for over two years. While talking to Patrick Debois at DevOpsDays Minneapolis 2014 the conversation meandered outside the typical confines of dev & ops. It was around the time Patrick joined Small Town Heroes, and I was reflecting on the parallels between themes at DevOpsDays and my work at Critical Mass. It felt like the issues we as a community were discussing were much more easily reconciled than the approach differences of Creative and Technology in my work environment.

I have come to believe that the premise that dev and ops are at odds is a fallacy. Sure, there are perverse organizational structures that silo expertise and incent them against one another, but I’m operating under the assumption that such easily identified wrongs aren’t present. What I was left to ponder (and later enumerate) were strategies and tactics that can be used to create high-performance, cohesive, multi-disciplinary teams.

Here’s an abridged outline of what I covered in the talk:

Mission statement
Participating & sharing in each others’ work process
Building trust
Surprise reduction
Psychological safety

"Understand your assumptions and
anticipate other's needs."

DevOps is about tools and culture. As much as I enjoy the occasional tool-smithing indulgence, I find the topic of leadership’s role in creating an environment that nurtures high-performance teams even more enjoyable and satisfying. Leave a comment or ping me on Twitter if you’d like to discuss further. Cheers.

https://speakerdeck.com/lanyonm/creative-and-technology-a-partnership-devopsdays-msn-2016

Log Aggregation with Log4j, Spring, and Logstash

Tue, 29 Dec 2015 00:00:00 +0000

While parsing raw log files is a fine way for Logstash to ingest data, there are several other methods to ship the same information to Logstash. These methods each have trade-offs that may make them more or less suitable for your particular situation. I have posted about multiline tomcat log parsing before, and this post is an attempt to compare that and other methods I’ve explored: log4j as JSON, log4j over TCP, and raw log4j with the multiline codec.

These examples were developed on one machine but are designed to work in an environment where your ELK stack is on a separate machine/instance/container. In the multi-machine environment Filebeat (formerly logstash-forwarder) would be used in cases where the example uses the file input.

For posterity’s sake, these are the software versions used in this example:

Java 7u67
Spring 4.2.3
Logstash 2.1.0
Elasticsearch 2.1.1
Kibana 4.3.1

I first began to author this post on Nov, 28th 2014, so please forgive any options presented that are no longer in favor. Also, you’ll notice that slf4j is used as an abstraction for log4j in the code samples.

Log4j As JSON

This method aims to have log4j log as JSON and then use Logstash’s file input with a json codec to ingest the data. This will avoid unnecessary grok parsing and the thread unsafe multiline filter. Seeing json-formatted logs can be jarring for a Java dev (no pun intended), but reading individual log files should be a thing of the past once you’re up and running with log aggregation. Also, you can run two appenders in parallel if you have the available disk space.

Instead of using a PatternLayout with a heinously complex ConversionPattern, let’s have a look at log4j-jsonevent-layout. The prospect of a solution that is entirely in configuration fits the bill, and I can live with yet another logging dependency.

Including the dependency in a pom.xml is quite easy:

<dependency>
    <groupId>net.logstash.log4j</groupId>
    <artifactId>jsonevent-layout</artifactId>
    <version>1.7</version>
</dependency>

The log4j.properties will look familiar, and I’ll explain UserFields in a bit.

log4j.rootLogger=debug,json

log4j.appender.json=org.apache.log4j.DailyRollingFileAppender
log4j.appender.json.File=target/app.log
log4j.appender.json.DatePattern=.yyyy-MM-dd
log4j.appender.json.layout=net.logstash.log4j.JSONEventLayoutV1
log4j.appender.json.layout.UserFields=application:playground,environment:dev

And the Logstash config is as you’d expect:

input {
  file {
    codec => json
    type => "log4j-json"
    path => "/path/to/target/app.log"
  }
}
output {
  stdout {}
}

The values set in the UserFields are important because they allow the additional log metadata (taxonomy) to be set in the application configuration. This is information about the application and environment that will allow the log aggregation system to categorize the data. Because we’re using the file input plugin we could also use add_field, but this would require separate file plugins statements for every application. Certainly possible, but even with configuration management more of a headache than the alternative. Also, the path parameter of the file plugin is an array so we can specify multiple files with ease.

If everything is set correctly, log messages should look like this in Kibana:

A log message from Playground using log4j-jsonevent-layout

As you can see, the UserFields are parsed into Logstash fields. If you prefer these values to be set via command line and environment variable, the library provides a way that will override anything set in the log4j.properties.

Log4j over TCP

This method uses log4j’s SocketAppender and Logstash’s log4j input. Log events are converted into a binary format via the SocketAppender and streamed to the log4j input. The advantages here are that the new log4j appender can be added without additional dependencies and that we are able to avoid dealing with the multiline filter. Let’s look at the implementation before digging into the shortcomings.

Here’s a snippet of the log4j.properties:

log4j.rootLogger=debug,tcp

log4j.appender.tcp=org.apache.log4j.net.SocketAppender
log4j.appender.tcp.Port=3456
log4j.appender.tcp.RemoteHost=localhost
log4j.appender.tcp.ReconnectionDelay=10000
log4j.appender.tcp.Application=playground

And the corresponding snippet of Logstash config:

input {
  log4j {
    mode => "server"
    host => "0.0.0.0"
    port => 3456
    type => "log4j"
  }
}
output {
  stdout {}
}

One of the log4j configurations above that you rarely see is Application. When this parameter is set Logstash will parse it into an event field. This is handy, but may not satisfy your logging taxonomy - exposing one of this method’s shortcomings: tagging log events with application and environment identifying information. The way this is typically done in the Logstash config is with add_field on an input plugin. Taking the typical approach would mean a different input plugin/port for each java app sending logs - not fun to manage at scale!

Mapped Diagnostic Context

Mapped Diagnostic Context (MDC) provides a way to enrich standard log information via a map of values of interest. Thankfully the log4j plugin will parse MDC hashes into log event fields. The MDC is managed on a per-thread basis, but a child thread automatically inherits a copy of the MDC from it’s parent. This means that log taxonomy can be set to the MDC in the application’s main thread and affect every log statement for its entire lifespan.

Here’s minimalistic example:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

public class LoggingTaxonomy {

    private static final Logger log = LoggerFactory.getLogger(LoggingTaxonomy.class);

    static public void main(String[] args) {
        MDC.put("environment", System.getenv("APP_ENV"));
        // run the app
        log.debug("the app is running!");
    }
}

With a PatternLayout conversion pattern like %d{ABSOLUTE} %5p %c{1}:%L - %X{environment} - %m%n and APP_ENV set to dev you’d expect to see a log statement like:

15:23:03,698 DEBUG LoggingTaxonomy:34 - dev - the app is running!

How and where to integrate the MDC values will vary widely based on the framework used by the application, but every framework I’ve ever used has an appropriate place to set this information. In Spring MVC with Java Config it can go in one of the AppInitializer methods. There are additional uses for MDC, which I’ll write about in a future post.

Once all the code and config is correct, the enhanced logs will flow into your Kibana dashboard like so:

A log message from Playground using Log4j over TCP and MDC for additional log event fields

A few things to note about this approach:

The logging level is stored in priority, not level as is with log4j-jsonevent-layout
There is no source_host field, so you may need to add that via MDC as well

Raw Log4j and the Multiline Codec

My multiline parsing post used the multiline filter plugin, but as mentioned above that plugin isn’t threadsafe. I wanted to provide a slight update to that approach that uses the multiline codec instead of the filter. I’ve modified the original example as little as possible and integrated the relevant bits into the playground app.

Here’s a snippet of the log4j.properties:

log4j.rootLogger=debug,file

log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.File=target/file.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS ZZZ} | %p | %c - %m%n

And the corresponding snippet of Logstash config (the grok pattern file can be found here):

input {
  file {
    type => "raw-file"
    path => "/path/to/target/file.log"
    add_field => {
      "application" => "playground"
      "environment" => "dev"
    }
    codec => multiline {
      patterns_dir => "/path/to/logstash/patterns"
      pattern => "(^%{TOMCAT_DATESTAMP_PATTERN})|(^%{CATALINA_DATESTAMP_PATTERN})"
      negate => true
      what => "previous"
    }
  }
}
output {
  stdout {}
}

You’ll notice that I used add_field to add application and environment fields because adding those to the ConversionPattern and grok parsers would’ve required some heavy lifting. If I’d built-out this solution fully, I would have integrated these via MDC as described above and made the ConversionPattern and Grok parse updates.

The log message will look like this in Kibana:

A log message from Playground using the file input and multiline codec to parse a raw Log4j

In my opinion there are several shortcomings to this approach:

Creating multiline parsers can be tough.
Grok parse patterns are tightly coupled to Conversion pattern and require adjustments in both places for changes.
Developers won’t be able to add MDC information and have it automagically show up in the log aggregation system.

Other Options

Log4j isn’t the only logging solution for Java. Logback is growing in popularity and implements the slf4j API making it swappable with Log4j or JUL. The logstash-logback-encoder looks particularly robust. If you’re coming from a Log4j implementation be sure to use the LogstashTcpSocketAppender, not the LogstashSocketAppender. The latter uses UDP and debugging an incident where log messages may have been dropped is a recipe for disaster. Given more time this would be my next exploration.

Summary

I hope the comparison of these methods is helpful. If it’s not already clear, my preference is log4j as JSON using log4j-jsonevent-layout and Filebeat. Logstash-forwarder or Filebeat is already on many of our servers, so this is an easy approach for us. I’m interested to hear others’ experiences managing their log aggregation pipeline.

Continuous Security with OWASP's Dependency Check Maven Plugin

Tue, 22 Dec 2015 00:00:00 +0000

On the heels of the previous post about continuously delivering documentation I wanted to show how easy it is to integrate dependency vulnerability checks and reports into a Maven-based delivery pipeline. The OWASP Top 10 2013 contains an entry about Using Components with Known Vulnerabilities, so being able to check project dependencies against a canonical list of vulnerable libraries is critical to compliance with the Top 10. The OWASP Dependency Check utility uses NIST’s National Vulnerability Database (NVD) to identify the vulnerable dependencies, so the list is always up-to-date.

The Dependency Check utility is conveniently wrapped by the dependency-check-maven plugin. The usage information contains several examples and the configuration allows plenty of tuning for the analyzer. With a single configuration parameter the plugin can fail a build if a vulnerable dependency is found.

CVSS Scores and the POM

The Common Vulnerability Scoring System (CVSS) uses several aspects of exploit-ability and impact to determine a base score. Each vulnerability in the NVD has a base score attached, and anything above 4.0 is considered Medium or High severity (CVSS overview).

Events following the publishing of a CVE can affect the CVSS score (e.g., an exploit kit is published). These adjusted scores are referred to as temporal scores, and CVSS provides a calculator to help understand the effect. The calculator is also quite handy to help understand what influences a score.

Thanks to the examples, configuring the POM is quite easy. Here’s the configuration I’m using in my pom.xml:

<project>
    <build>
        <plugins>
            <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>1.3.3</version>
                <configuration>
                    <cveValidForHours>12</cveValidForHours>
                    <failBuildOnCVSS>4</failBuildOnCVSS>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

The configuration above will fail the Maven run if a dependency with a CVSS score above 4 is found and will recheck the NVD only every 12 hours. To run a standalone dependency check:

mvn dependency-check:check

Report Integration

Integrating a dependency check report into the generated Maven site is also quite easy thanks to the examples. Here’s the relevant snippet of the pom.xml:

<configuration>
    <reportPlugins>
        <plugin>
            <groupId>org.owasp</groupId>
            <artifactId>dependency-check-maven</artifactId>
            <version>1.3.3</version>
            <configuration>
                <name>Dependency Check</name>
            </configuration>
            <reportSets>
                <reportSet>
                    <reports>
                        <report>aggregate</report>
                    </reports>
                </reportSet>
            </reportSets>
        </plugin>
    </reportPlugins>
</configuration>

A “Dependency Check” link will be added to the Project Reports page of the generated site that points to a page that looks like this:

OWASP's Dependency Check showing CVE-2015-6420 in Apache's commons-collections:3.2.1

For illustrative purposes I included a known vulnerable dependency. Running mvn site didn’t cause the build to fail because the plugin configuration used during the site lifecycle phase is separate from what was defined for the standalone check.

Summary

While not a full solution for security in your continuous delivery process, the dependency-check-maven plugin is a great way for a Java project to check the A9 box on the OWASP Top 10. Additionally, the visibility that the report gives dependency security is fantastic.

One handy feature I didn’t mention is that a suppression list can be used to ignore positives. This can be used for false positives, or to ignore acknowledged vulnerabilities currently awaiting resolution in the backlog. This would allow the CI process to continue to run while the project is brought into compliance.

Publishing a Maven Site to GitHub Pages with Travis-CI

Sat, 19 Dec 2015 00:00:00 +0000

Part of continuous delivery is continuously delivering documentation along with the software. I’d go so far as to argue that it’s part of the software. In the past I’ve had to remember to run mvn site periodically to ensure that the latest javadoc and dependency updates report get created, but there’s gotta be a better way!

The key to this integration is having Travis authenticate back to GitHub to publish to gh-pages in a secure way. The default suggestion for publishing with GitHub’s site-maven-plugin is to add your username & password or an oauth token to ~/.m2/settings.xml. This won’t work for Travis-CI and would leak the oauth token.

POM Configuration

The site-maven-plugin configuration is exactly the same as GitHub’s example, but that’s to be expected. The important configuration is to allow the oauth token to be read from an environment variable (excerpt from pom.xml):

<project>
    <properties>
        <github.global.server>github</github.global.server>
        <github.global.oauth2Token>${env.GITHUB_OAUTH_TOKEN}</github.global.oauth2Token>
    </properties>
</project>

To be able to run mvn site locally you’ll need to run export GITHUB_OAUTH_TOKEN="your-github-personal-access-token" or add that line to your dotfiles. To create the token follow these instructions. The token I created has repo and user:email access.

Travis-CI Configuration

Encrypted Environment Variable

Getting your GitHub token into the Travis-CI environment var is very well documented. Here’s a handy copy/paste of the command you’ll want to run to encrypt your environment variable:

travis encrypt GITHUB_OAUTH_TOKEN="your-github-personal-access-token" --add env.global

The one thing not covered in the Travis-CI docs page is that you can have multiple encrypted environment variables like so:

env:
  global:
    secure: bigEncryptedString/One
    secure: bigEncryptedString/Two

Build Timeout

Another thing to consider is that the site-maven-plugin can take quite a while to prepare and upload the site html to your gh-pages branch. By default this process does not generate log statements and Travis-CI may time-out. You can either have Travis extend the timeout or enable debug logging for Maven: mvn site -X. I chose the second option because I didn’t want some other issue to cause long build times. The .travis.yml has the full details.

Summary

You can see this working in my Spring playground repository which publishes here. Please let me know if you find this helpful or have suggestions for improvements.

ChatOps: Pingdom Alerts Pushed into HipChat with AWS Lambda and API Gateway

Wed, 25 Nov 2015 00:00:00 +0000

You probably searched “pingdom alerts in hipchat” or “pingdom hipchat integration” and were unhappy to find that there’s no direct method to integrate the two services. I was too - but it gave me the chance to use the AWS API Gateway and AWS Lambda to connect the two services. I assume you’re relatively familiar with the functionality that API Gateway and Lambda provide as well as getting-started experience with Node.js.

Pingdom Webhooks

The documentation on the Pingdom site describes how to find the webhook payload structure, so I followed their advice and observed the up & down webhook events. I don’t use it this demo, but each of the webhooks a X-Request-Id header was sent with a value like 6645779c-18a9-473d-808e-2b74450c7347. You may find this useful for tracing purposes.

Down

As you can see, the assign webhook is a GET request with the message payload as an URI encoded json string. It would be nice if this was a POST, but ¯\_(ツ)_/¯.

GET /webhook-endpoint?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22assign%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22down%22%7D

The decoded querystring looks like:

message = {
  "check": "1834565",
  "checkname": "just a test",
  "host": "www.example.com",
  "action": "assign",
  "incidentid": 8765,
  "description": "down"
}

Up

Aka notify_of_close or resolved:

GET /webhook-endpoint?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22notify_of_close%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22up%22%7D

message = {
  "check": "1834565",
  "checkname": "just a test",
  "host": "www.example.com",
  "action": "notify_of_close",
  "incidentid": 8765,
  "description": "up"
}

Testing with curl

For testing purposes you may want a quick curl statement to act as Pingdom (so you don’t have to deliberately take a monitored endpoint down):

curl -H 'X-Request-Id: 6645779c-18a9-473d-808e-2b74450c7347' https://1x1x1x1x1x.execute-api.us-east-1.amazonaws.com/prod/pingdom-webhook?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22assign%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22down%22%7D

Now that we understand Pingdom’s webhook a bit better, let’s have a look at the AWS parts.

AWS Lambda

Before we configure the API Gateway, let’s have a look at the Lambda function. We do this first because you’ll need to select the Lambda when you create the API Gateway resource. I chose to use Node.js, but the code is straightforward and should be able to be ported to python easily.

When creating a Lambda function you’ll be prompted to select a blueprint. Any will do, but microservice-http-endpoint will most closely mirror the functionality of our Lambda. With a blueprint selected, you’ll need to configure the name, runtime, handler (entry point), IAM role, memory, and timeout for the Lambda. You will likely need to create a basic IAM role to allow your Lambda (the AWS console will help you do this), and you’ll want to decrease the memory requirement to 128MB.

The entry point for the Lambda in this example is index.pingdomToHipchat, and per the Lambda spec the function takes the event and context objects. The full repo is on GitHub, and I’ve included the index.js below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
'use strict';

var http = require('https');
var config = require('./config.js');

exports.pingdomToHipchat = function(event, context) {
  // the contents of event is dependent on the configuration of API Gateway
  // console.log('the message is', event.message);

  // there some things that the decodeURI method doesn't clean up for us
  var msg = JSON.parse(decodeURI(event.message).replace(/\+/g, ' ').replace(/%3A/g, ':').replace(/%2C/g, ','));
  console.log('the message json is:\n', msg);

  var hc_msg = {
    color: msg.description === 'down' ? 'red' : 'green',
    message: msg.checkname + ' is ' + msg.description + ' (' + msg.host + ')',
    notify: false,
    message_format: 'text',
  };

  console.log('hipchat message:\n', hc_msg);

  var http_opts = {
    host: 'api.hipchat.com',
    port: 443,
    method: 'POST',
    path: '/v2/room/' + config.hipchat.room + '/notification?auth_token=' + config.hipchat.token,
    headers: {
      'Content-Type': 'application/json',
    }
  };

  var req = http.request(http_opts, function(res) {
    res.setEncoding('utf8');
    res.on('data', function (chunk) {
      console.log('BODY:', chunk);
    });
    res.on('end', function () {
      if (res.statusCode === 204) {
        console.log('success - message delivered to hipchat');
        context.succeed('message delivered to hipchat');
      } else {
        console.log('failed with', res.statusCode);
        context.fail('hipchat API returned an error');
      }
    });
  });

  req.on('error', function(e) {
    console.log('problem with request:', e.message);
    context.fail('failed to deliver message to hipchat');
  });

  req.write(JSON.stringify(hc_msg));
  req.end();
};

On line 11 event.message is decoded and then cleaned further to compensate for the remaining encoding weirdness. Once I figured out how to translate the querystring params in API Gateway, iterating on this was the last bit of magic to get a json representation of the Pingdom alert in the Lambda function. The rest of the code creates the HipChat notification API request & payload, and then sends the request.

The console.log statements are sent to CloudWatch - which can help you audit or debug during development. The built-in Lambda test functionality is also captures this output, and is the quickest way to ensure that changes to the Lambda function as expected. This is the test event for the Lambda above:

{
  "requestId": "6645779c-18a9-473d-808e-2b74450c7347",
  "message": "%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22notify_of_close%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22up%22%7D"
}

Lastly, if you want to be able to copy/paste this code into the AWS Lambda console for testing you’ll need to remove the config.js require on line 4 and replace the two values on line 27 with your HipChat token and room id. More on finding these values for your setup below.

Now that we have the Lambda squared away, let’s see how it gets wired up with the API Gateway.

AWS API Gateway

The API Gateway takes the Pingdom GET request and populates the event object passed to the Lambda. This step would be much easier if the Pingdom webhook used POST instead of GET, but you’ll learn something interesting about the API Gateway as a result.

Inside the API Gateway console create a new API, create a resource, and create a GET method. When creating the method, you’ll need to select “Lambda Function” as the integration type the region the Lambda function is deployed into, and the Lambda name (which will auto-complete).

At this point you should see something like this:

The API Gateway method before configuration.

There’s a couple things we need to do to translate the incoming Pingdom GET into the event that the Lambda expects. The first is to define the method request. We need to specify the X-Request-Id header and message query string as shown below:

The API Gateway method request configuration.

The magic happens in the integration request configuration. Select “Lambda Function” for the integration type, select your Lambda function name, and add an application/json content-type Mapping Template. The previous step made X-Request-Id and message available to be mapped into the event as follows:

The API Gateway method integration request configuration.

Header parameters and query string parameters are both fetched via the "$input.params('key')" function. Due to the encoding of the Pingdom webhook, we need to urlEncode the message value to avoid a parse exception.

{
  "requestId": "$input.params('X-Request-Id')",
  "message" : "$util.urlEncode($input.params('message'))"
}

Once all this is configured, you’ll need to “Deploy API”. Stages are used as environments, so you can test API changes as they roll from development to production. If you make changes you’ll need to redeploy the API. Assuming your production environment is prod, you’ll receive a url like https://1x1x1x1x1x.execute-api.us-east-1.amazonaws.com/prod/pingdom-webhook.

Prepping HipChat

HipChat’s v2 API requires you to create an integration to push notifications to rooms. This is done via the HipChat admin console and creates an authentication token for the room id specified.

If you clone the repo, you’ll want to cp config.js.sample config.js and add your token and room id to the config:

module.exports = {
  hipchat: {
    token: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    room: '1111111'
  }
};

You can find more detail about how to package multiple files for upload to a Lambda function in the README.

Putting it all together

If all goes well, you should be able to issue curl command above and see a notification appear in the configured HipChat room. Use down and up in the description to see red and green highlighted messages:

What you should see in HipChat if everything goes well.

In Pingdom you’ll need to go into Alerting > Alerting Endpoints and add a webhook contact method to an Alerting Endpoint used by an Alert Policy that is used by the check you’d like to see in HipChat. You likely already have an Alert Policy used for your check, so adding an additional endpoint for that policy should be straightforward.

I hope this works for you, and please let me know if it doesn’t! Big thanks to @ripienaar for his post on translating webhooks that got me thinking about this in the first place.

ChatOps: Hubot Grafana Images in HipChat

Wed, 30 Sep 2015 00:00:00 +0000

As we continue toward ChatOps and making our work visible at work, the next phase of maturing our monitoring systems is to create a query-able interface to our visualization system (Grafana) from HipChat. Grafana is a system I’ve become quite fond of and helped author the Chef cookbook for. The HTTP API for Grafana has matured, and the time seemed right to create this integration.

High-level Design

Having read about Librato’s ChatOps and seen Etsy’s nagios-herald, I had a rough idea of the user experience I wanted. With a head full of hindsight bias, here are some of the requirements:

A user-friendly query interface in chat (no magic numbers, server-specific names, etc.)
Images posted should be available in chat without additional authentication
Able to utilize our existing Grafana server

I was delighted to find that Stephen Yeargin had already written hubot-grafana, a script that did all the heavy lifting for the first requirement. The Grafana docs site also has a how to integrate Hubot with Grafana article. Stephen’s hubot script provides for discovery of dashboards, per-panel queries, template variables, and time-range queries. It’s really quite fantastic. However, it assumes that S3 will be used to host the images. While that’ll work for most folks (and certainly could work for us), I wanted to be able to use our existing Grafana server to house this integration. To achieve this I had to modify grafana.coffee. More on that below.

The default configuration provided by the chef-grafana cookbook includes Nginx as a proxy for grafana-server. For work we wrap the community cookbook to configure TLS, LDAP, and Grafana’s datasources. It seemed like a natural extension of visualization’s responsibility to have a small app on the Grafana node that can fetch/save rendered panel images and then use Nginx to serve those images. I called that small application grafana-images. More on that below as well.

Modifications to hubot-grafana

As mentioned above, I had to modify the hubot-grafana script to provide an alternate image persistence method (alternative to S3). The coffeescript additions are relatively straightforward:

customFetchAndUpload = (msg, title, url, link) ->
  requestHeaders = {
    encoding: "utf8",
    Authorization: "Bearer #{grafana_api_key}",
    Accept: "application/json"
  }

  req_opts = {
    method: "POST",
    url: "#{grafana_images_host}/grafana-images",
    headers: requestHeaders,
    json: {
      imageUrl: url
    }
  }

  # post to grafana-images
  request req_opts, (err, res, json) ->
    robot.logger.debug "grafana-images POST: #{req_opts.url}, content-type[#{res.headers['content-type']}]"

    if res.statusCode == 200
      sendRobotResponse msg, title, json.pubImg, link
    else
      robot.logger.debug res
      robot.logger.error "Upload Error Code: #{res.statusCode}"
      msg.send "#{title} - [Access Error] - #{link}"

The Grafana API key is provided to the script by an environment variable and the newly added environment variable HUBOT_USE_GRAFANA_IMAGES determines whether or not to use the customFetchAndUpload code-path. The full diff can be found here.

As you can see, the /grafana-images uri is hard-coded. That’s because the route used by grafana-images is hard-coded. Also, note that the necessary authentication token is passed along with the json payload. In many ways this function is treating grafana-images as a proxy for Grafana.

Another addition to note is the help text I added to the hubot script. You can ask the bot “graf help” and it’ll respond with increasingly complex query samples. Yay for user friendliness!

grafana-images

Following my experience with http-stats-collector, Golang seemed like a good choice for the small application. It acts as a proxy and therefore expects only two things: a valid API token and json payload containing the full Grafana panel render url. To give more context to what’s happening, here’s an http call diagram:

The http calls required for hubot-grafana using grafana-images numbered by sequence

The customFetchAndUpload function described above is call #4. From there grafana-images will fetch (#5 & #6), save, and return a sharable image url via json (#7). Here’s a snippet from grafana-images’ handlers.go:

// Fetch image
var client = http.Client{}
req, err := http.NewRequest("GET", image.Url, nil)
req.Header.Add("Accept", "application/json")
req.Header.Add("Authorization", token)
resp, err := client.Do(req)
if err != nil {
  log.Fatalf("http.Get -> %v", err)
  w.WriteHeader(500)
  return
}
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
  log.Fatalf("ioutil.ReadAll -> %v", err)
  w.WriteHeader(500)
  return
}

// Save image
fileName := fmt.Sprintf("%x.png", md5.Sum(data))
resp.Body.Close()
err = ioutil.WriteFile(fmt.Sprintf("%s/%s", imagePath, fileName), data, 0666)
if err != nil {
  log.Fatalf("ioutil.WriteFile -> %v", err)
  w.WriteHeader(500)
  return
}

// Return image location
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, "{\"pubImg\":\"%s/%s\"}", imageHost, fileName)

There are several variables assumed to be set:

image - the requested imageUrl from the json
token - the contents of the Authorization header
imagePath - a path on disk to store the saved images
imageHost - the host used in the building the json response

If everything is configured correctly, the Grafana dashboard panel will be saved to disk and the json sent back to hubot-grafana. Further detail can be found on GitHub. I tired to make all the error messages helpful and actionable, but if you find an error condition that isn’t well explained, please open a GitHub issue.

Security

You may have noticed that the app very simply downloads whatever is specified at imageUrl and saves it as a png. This can be dangerous given that nothing checks to ensure that the contents are in-fact an image and not an exploit. Take care to only allow specific traffic to make requests of grafana-images. I may add a check via Golang’s png package to ensure proper encoding, but it may be quite some time before that happens (pull requests welcome).

Nginx Config

As mentioned above, I used Nginx to proxy grafana-server. I also use it to proxy grafana-images and serve the saved panel images. Here’s a sample conf that should would for this purpose:

# the grafana-server config has been omitted
upstream grafana-images {
  server 127.0.0.1:8080;
}

server {
  # excerpt for grafana-images
  location /grafana-images {
    proxy_pass http://grafana-images;
    allow 10.0.0.11; # ip of server running hubot
    deny all;
  }
  location /saved-images {
    root /opt;
  }
}

Note that the imageHost passed to grafana-images is the FQDN plus the location of the saved images. The value used will be dependent on the web server hosting the saved images.

Other Uses

Because grafana-images exposes its functionality over a simple HTTP API, expanding its purpose should be straightforward. The app expects an "Authorization: Bearer grafana-token-goes-here" header and a json payload:

{
  "imageUrl": "https://grafana.example.com/render/dashboard-solo/db/sample-dashboard/?panelId=5&width=1000&height=500&from=now-6h&to=now&var-server=test-server"
}

Sensu Notifications

At work we have incorporated Grafana panel image embedding functionality into our Sensu HipChat handler. We started with the Sensu community HipChat handler and modified the message body heavily for our purposes.

The code to add Grafana panel images to Sensu HipChat notifications is roughly:

if @event['check']['graph_image']
  # do some work to get the static image from grafana-images
  uri = URI.parse('https://grafana.example.com/grafana-images')
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  request = Net::HTTP::Post.new('/grafana-images')
  request.add_field('Content-Type', 'application/json;charset=utf-8')
  request.add_field('Authorization', 'Bearer grafana-token-goes-here')
  request.body = { 'imageUrl' => "#{@event['check']['graph_image']}&from=now-6h&to=now" }.to_json

  begin
    response = http.request(request)
    public_image = JSON.parse(response.body)['pubImg']
    message << "<br /><a href=\"#{public_image}\"><img src=\"#{public_image}\" /></a><br />"
  rescue StandardError => e
    message << " - [graph_image fetch failed (#{e})]"
  end
end

The @event['check']['graph_image'] value is assumed to be a valid dashboard panel render url without the from/to times: https://grafana.example.com/render/dashboard-solo/db/sample-dashboard/?panelId=5&var-server=test-server&width=1000&height=500. The panelId is can be obtained from the UI of the dashboard.

We manage our infrastructure with Chef and it creates all the Sensu checks, thus allowing us to programmatically build the checks. We add a graph_image attribute to the check that contains a panel render url associated with the metric(s) that can help provide context to the Sensu notification. Chef can give the FQDN of the Grafana node as well as the values for template attributes, so it all comes together quite cleanly.

Other Considerations

One thing not handled by grafana-images is saved image retention. You’ll need to create a purge policy that works for you. Once I’ve figured out how we’re going to handle that, I’ll add it here. :)

A Participant's Conference - DevOpsDays Chicago 2015

Tue, 01 Sep 2015 00:00:00 +0000

Last week was DevOpsDays Chicago 2015, my second DevOpsDays Chicago as a co-organizer. We learned a lot in our first year, and I feel that we improved substantially this time around. The anecdotal feedback over the past several days has been overwhelmingly positive, and it’s bolstering to feel that outpouring of appreciation after putting in so much effort. My experience as an organizer was much improved as well. We avoided bike-shedding as much as possible, more clearly divided responsibilities, and did our best to improve on our shortcomings in 2014.

There are most certainly aspects of the event that we can and will continue to improve upon. We have started the post-mortem process and will be publishing both the 2015 and 2014 post-mortem documents. Keep an eye out for that.

I wasn’t planning on typing up my thoughts about the conference, but a post by Carolyn Van Slyck has been on my mind…

Participation

"DevOpsDays is a participant's conference"

I’m not sure where I picked up the idea, but I recall saying it in our organizer Slack channel during the run-up to the 2014 event. It is not a platitude. It is at the heart of what makes DevOpsDays an amazing conference series. Open Spaces are typically a new experience for first-time participants and sometimes even dreaded due to the apparent chaos of unstructured time, but every anecdote I heard this year about Open Spaces was it being the best part of the conference.

It was my honor and responsibility this year to open the conference. It was with incredible pride that I was able to ask everyone to look down at their badge and see that they are a participant at DevOpsDays, to say that the DevOpsDays community values respect, inclusiveness, and diversity, to say that DevOpsDays is a safe space and remind everyone to be cognizant, and finally to say that whether or not they realized it at the time, their contributions to the community would be valued and respected.

These statements are backed by our Code Of Conduct. Originally introduced by DevOpsDays Pittsburgh, this code has become the template used by DevOpsDays events. It conveys a strong belief that is strongly held. As a participant I have spoken up when I have felt the code is violated. As an organizer, it was and is a guide for the tone to set.

I also introduced Open Spaces, and while I’ll never hold a candle to Patrick’s introduction I tried to extend the tone of respect and inclusiveness into the four principles of Open Spaces:

Whoever comes is the right people
Whatever happens is the only thing that could have
Whenever it starts is the right time
When it’s over, it’s over

I heard these statements repeated throughout the afternoon as if to confirm that it is indeed okay to leave an Open Space you’re no longer engaged with or continue an Open Space that isn’t over. Yes, absolutely.

Reflection

Experiences like Carolyn’s are what I and my co-organizers had hoped to nurture. A supportive community that isn’t just open to dialog, but actually encourages communication and sharing. At the end of DevOpsDays Chicago 2014, I felt like I’d submitted a huge PR to critical OSS that I’d been using for years - exhausted, but happy. This year, I feel pride of stewardship and for being part of something that is helping make peoples’ lives better.

Thanks

I’d like to thank the DevOpsDays Chicago 2015 participants, speakers, sponsors, volunteers, and support folks. The community that we convene for just a couple days is why I wanted to organize once again this year. I’d also like to thank my co-organizers. It wouldn’t be nearly as much fun if you weren’t all awesome.

The DevOpsDays Chicago 2015 Organizing Team

Web Performance Monitoring - DevOpsDays MSP 2015

Wed, 08 Jul 2015 00:00:00 +0000

I initially intended to speak about how we use configuration management to automate our real user measurement (RUM), but as I was putting together my talk I discovered that the real insight was how the same tools can be used to facilitate the front-end developer’s relationship with prod.

I opened the Ignite by speaking about the lack of operational visibility into the end user’s experience and how our initial attempt to close this gap was to use the Navigation Timing data. We use a Golang program called http-stats-collector to collect NavTiming data as well as javascript errors and CSP reports. Looking at the data after it was processed by our monitoring pipeline, we saw that the real insight was how javascript errors could tell the story of the end user’s experience - something NavTiming data did not do for us.

I concluded the talk with a sampling of the systems and templates we have in place to facilitate the collection of data. While this understanding of the end user experience is valuable, the data collection and visualization needs to be turn-key for it to be used across our organization.

https://speakerdeck.com/lanyonm/web-performance-monitoring

https://youtu.be/ku3O4HnMXrM?t=395

All photo credits including those in the og:metadata go to Bridget Kromhout or DevOpsDays Minneapolis 2015.

Here’s a transcript of the talk:

Hi, my name is Mike Lanyon, and I work at a digital ad agency called Critical Mass. I would like to talk about web performance monitoring. In DevOps we often talk about monitoring of the systems that provide experiences, but not the monitoring of the end user’s experience itself.
Critical Mass is a digital experience design agency. That means that we put the customer at the center of our process, through strategy, design, technology and analytics. We want to use our influence with clients to make their customers’ lives better.
More specifically than web performance, I’d like to talk about a front end developer’s relationship to prod. In years past this relationship may have been through an FTP client, but no-matter the technology there was a gap in operational visibility.
The front-end developer’s preparation for production has gotten quite robust. Run sass, concat & minify js. Maybe there’s a CDN, but then it gets a bit fuzzier… Where is the operational visibility in this?
WebPage test is a tool our teams use to quantify the end user’s experience. This chart shows the composite Speed Index of one of our web pages. The Index is the integral of the space above the charted lines, which represent the perceived completion of the page load.
But still there is no operational visibility. There’s no way for a web developer on my team to understand the performance of a user currently visiting our site. There’s nothing analogous to the app log with a stack trace detailing the user’s crummy experience.
RUM - real user monitoring. This is something that began to pop up a few years ago. This is different than synthetic monitoring because you’re collecting data from the experience of real users.
Navigation Timing API is one of the most well established means to collecting the real user’s performance. There are several solutions that will capture this information for you like Google analytics or NewRelic.
Golang. I toyed around with Go a couple years ago, and didn’t really understand how it could be useful to me at the time - but it’s reputation for being fast seemed like a good fit for collecting RUM performance data.
I created https-stats-collector. Inspired by a GDSTeam project called event-store that saves content security policy reports. I don’t really want to focus much on the Golang code here - only to highlight that it’s open source and would love to have some feedback on the project.
There are three primary routes offered by the application: nav-timing, js-error, and csp-report. In addition to nav-timing, I’ll talk about js-errors, which is fed by a global javascript error and logs the errors experienced by users.
As you can see, after nav-timing data is processed by our monitoring pipeline we’re able to create pretty, squiggly line graphs. I really enjoy these, and I think they’re great, but they don’t help tell the end user’s story.
This is screencap of Kibana showing the errors captured by the js-errors handler and processed by our ELK stack. When one of the senior developers first saw this, he pointed to a cluster of errors experienced by a single user and said, “I wonder what the story is there”.
We have found that while nav-timing data is really cool, javascript error reporting is the game changer. It gave the teams a level of traceability they’d never had before and equated to a front-end version of the app log.
This capability is really valuable, but it has to be operationalized and become part of the default path or path of least resistance.
We use Chef to put all the bits in place, http-stats-collector, statsd, logstash, elasticsearch, influxdb, kibana, grafana, etc. Configuration management also helps us standardize and scale the implementation.
Sample implementations that are framework agnostic help the client/product teams adopt the pattern and integrate it into their work.
We have gone so far as to make these RUM collection tools part of our standard stack. In this example a team asked for a simple webserver, and they got the webserver - but packed with all the data collection bits pre-installed.
If you take anything from this talk, please consider how to help create the connection between your technology teams and your users. Give your teams the tools to make this connection with prod - to form this relationship with the individual end user.
Thank you

Grafana Chef Cookbook Wrapper to Override Nginx SSL Template

Sun, 28 Jun 2015 00:00:00 +0000

I recently contributed to the overhaul and 2.0 release of the Grafana Chef cookbook. It was a nearly complete rewrite of the 1.x version, and many decisions were made along the way about what should (and should not) be included in the effort. The cookbook is designed to be as flexible as possible via attributes and to provide the user with a functional setup using the defaults. The previous version used Nginx as a web server, and it made sense to proxy the new Grafana with Nginx in the default setup.

One of the initially identified tasks was to provide SSL by default within the cookbook, but that proved to be foolish for two reasons: 1) it was well outside the scope of the Grafana cookbook and 2) SSL configs are highly dependent on several diverse factors ranging from web server version to client browser requirements. Creating a default Nginx SSL setup that was flexibly configurable with attributes was duly marked as out of scope, but not without some discussion.

For reference, here’s the default Nginx template from the Grafana cookbook:

template '/etc/nginx/sites-available/grafana' do
  source node['grafana']['nginx']['template']
  cookbook node['grafana']['nginx']['template_cookbook']
  notifies :reload, 'service[nginx]'
  mode '0644'
  owner 'root'
  group 'root'
  variables(
    grafana_port: node['grafana']['ini']['server']['http_port'] || 3000,
    server_name: node['grafana']['webserver_hostname'],
    server_aliases: node['grafana']['webserver_aliases'],
    listen_address: node['grafana']['webserver_listen'],
    listen_port: node['grafana']['webserver_port']
  )
end

Overriding The Nginx Conf Template

When defining an Nginx template with SSL enabled, it’s helpful to have additional variables passed to the template so info like cert locations can be more flexibly defined. Using cookbook and source attributes provided by the recipe would allow for the wrapper cookbook to define a new template, but not allow for additional variables to be passed to the template. By taking advantage of Chef’s compile phase, we can alter the template['/etc/nginx/sites-available/grafana'] resource to not only use the template of our choosing, but also pass additional attributes to the resource.

The resource definition below is from a Grafana cookbook wrapper recipe:

begin
  r = resources(template: '/etc/nginx/sites-available/grafana')
  r.cookbook 'wrapper-cookbook'
  r.source 'default/grafana/nginx.conf.erb'
  r.mode 0644
  r.variables(
    grafana_port: node['grafana']['ini']['server']['http_port'] || 3000,
    server_name: node['grafana']['webserver_hostname'],
    server_aliases: node['grafana']['webserver_aliases'],
    listen_address: node['grafana']['webserver_listen'],
    listen_port: node['grafana']['webserver_port'],
    ssl_cert_dir: node['wrapper-cookbook']['ssl']['cert_dir'],
    ssl_cert_name: node['wrapper-cookbook']['ssl']['cert_name'],
    ssl_key_dir: node['wrapper-cookbook']['ssl']['key_dir'],
    ssl_key_name: node['wrapper-cookbook']['ssl']['key_name'],
    ssl_dhparam_name: node['wrapper-cookbook']['ssl']['dhparam']
  )
  r.notifies :reload, 'service[nginx]', :immediately
  rescue Chef::Exceptions::ResourceNotFound
    Chef::Log.warn 'could not find template to override!'
end

The recipe is grafana and the containing cookbook is wrapper-cookbook. When compared to the template within the Grafana cookbook’s _nginx.rb, you can see that five additional SSL-related attributes are passed to the template (lines 12-16).

A Sample Nginx SSL Template

The nginx.conf.erb will be dependent on your SSL requirements, but here’s an example:

#
# This file was generated by Chef for <%= node['fqdn'] %>.
# Do not modify this file by hand!
#

upstream grafana {
  server 127.0.0.1:<%= @grafana_port %>;
}

server {
  listen                <%= "#{@listen_address}:" if !@listen_address.nil? && !@listen_address.empty? %><%= @listen_port %>;

  server_name           <%= @server_name %> <%= @server_aliases.join(" ") %>;
  access_log            /var/log/nginx/<%= @server_name %>.access.log;

  ssl on;
  ssl_certificate <%= @ssl_cert_dir %>/<%= @ssl_cert_name %>;
  ssl_certificate_key <%= @ssl_key_dir %>/<%= @ssl_key_name %>;
  ssl_dhparam <%= @ssl_cert_dir %>/<%= @ssl_dhparam_name %>;

  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;

  add_header Strict-Transport-Security max-age=31536000;

  ssl_stapling on;
  ssl_stapling_verify on;

  location / {
    proxy_pass http://grafana;
  }
}

Your mileage will most certainly vary. If you’re looking for suggestions on what SSL configs to use Mozilla has put together a great TLS configuration generator: https://mozilla.github.io/server-side-tls/ssl-config-generator/.

Conclusion

You could argue that this is unnecessary for the Grafana cookbook’s _nginx.rb given that the recipe is so simple, but it illustrates the power of Chef’s compile phase to override the defaults of the cookbooks used.

In addition to a default Nginx setup, the cookbook provides LWRPs for creating datasources, dashboards, organizations, and users. It’ll be exciting to see how people use and extend it.

Sample Dashboard configured with the Grafana cookbook.

Innovation in a Creative Company - DevOpsDays NYC 2015

Thu, 30 Apr 2015 00:00:00 +0000

I had the privilege of giving an Ignite at DevOpsDays New York 2015 this past week. The theme of the event was inclusivity, complexity, and empathy, which really made for a great event.

My talk was about some of the successes and failures I’ve experienced when trying to bring DevOps values to my organization and explain how empathy, inclusion, and communication are integral to the successes. I discussed a handful of ideas, including surprise reduction, curiosity, and sharing, and briefly touched upon how diversity in teams enables stronger ideas and better innovation. The journey of cultural change will be different from one organization to the next, but I hope to ignite more conversation on this topic by talking about the changes we’ve attempted.

Ultimately, by working to understand other disciplines and finding empathetic, inclusive ways to bridge the gaps between “us and them,” technology can work more collaboratively with other disciplines, and we can create better, more innovative products as a result.

Here’s a transcript of the talk:

Hi, my name is Mike Lanyon, and I work at an ad agency called Critical Mass. We are a digital experience design agency with a relentless focus on the customer. We’re comprised of four primary disciplines: Strategy, Design, Technology and Marketing Science, and we work together to design experiences for large brands.
We are not a technology company, but as a digital agency, technology is essential to what we deliver. I like to say that software is the delivery mechanism for brand experiences. And our challenge is to have our disciplines work together in a way that delivers the most innovative yet intuitive experiences.
Creating a culture where the disciplines work together to innovate is difficult. Creating a Kata that we practice across the disciplines has been exceedingly difficult. I’m going to talk about some of the qualities that I think help lead us in the right direction.
If you’ve been part of the DevOps community for a while you’ll be familiar with lessons of Deming, Goldratt, and Shewhart. While I have found their teachings of statistical process control, system of profound knowledge and theory of constraints valuable, they alone are insufficient to unlocking innovation in our organization.
When we look at what drives and delivers value in our organization, we find communication, empathy and inclusivity. You may be thinking, “yeah, duh.” but I assure you that if it was as simple our theme at DevOpsDays New York wouldn’t be what it is.
One of the ways I look at communication is surprise reduction. People like to know what’s going on and what to expect. Secrecy is toxic. We’ve all experienced this, whether it’s not knowing enough context for an assignment or simply not knowing that a project is underway.
It seems intuitive to keep others in the loop, but surprise reduction is more than that. It’s about understanding what you already know that others may not, and anticipating their needs. This takes a level of effort that goes beyond common sense, but it’s well worth the energy.
When everyone is on the same page - it’s more likely that you’ll get diverse creative contribution from all disciplines. Innovation sprouts from this proactive communication.
I have found that curiosity is another key quality for nurturing innovation. When you’re curious you want to know how things work, you want to know what motivates people. I believe it to be an innate quality of intrinsically motivated people.
Curiosity has driven designers to learn how to use the web inspector. It’s driven information architects to understand how javascript and css power animations, and it drove me to learn about operations. These pieces on the screen where created by my cousin, Sarah Walker who happens to live and work in Brooklyn. These pieces were inspired by how technology has interwoven itself into our lives.
If there is one thing you take away from this talk, reflect on your curiosity and act on that interest. If you know someone with that talent or knowledge, ask them about it. Start a conversation.
Now put yourself on the other side of that interaction. When you’re in the position to answer questions or teach, please, please, please take the time to do so. Continuing the conversation will start a relationship that lasts longer than the dialog.
Early in my career, I didn’t value sharing. I wasn’t confident enough to share work-in-progress and documentation wasn’t valued until after it was needed. As I matured I came to understand how sharing enables the organization to gain knowledge as a whole.
I came to understand much more holistically that the creative process depends on sharing. I still struggle with how to keep the sharing feedback loop going within the creative process, but the Chicago voting mantra applies: do it early and often.
Communication, curiosity, and sharing ladder into a what I believe to be a higher order humanistic quality: empathy. Empathy is understanding and sharing in the feelings of another. And empathy is requisite to innovation. Good design is rooted in Empathy.
Imagine trying to design a product without understanding the motivation of the consumer - ok, that’s kinda unheard of, but consider the value of understanding and sharing in emotional needs of both your consumers and your team members.
At Critical Mass we not only consider how our software will be received by operators for example, but also think empathetically across discipline lines. Contextualizing a decision with an anecdote while not hiding the details helps the other disciplines relate to what we do in technology.
Inclusivity is not to be discounted. It too is requisite to innovation. Ensuring your culture is aware of exclusive behavior and discourages it will allow a more diverse set of contributors, which will ultimately lead to greater perspective and thereby innovation.
The creative process, supporting structures, and empathetic culture will never be available as a service or written as code. But we can most certainly apply the values that we hold dear in the DevOps community to this challenge and succeed.
Thank you.