[{"content":"","date":null,"permalink":"/tags/cams/","section":"Tags","summary":"","title":"CAMS"},{"content":" \u0026ldquo;The premise that dev and ops are at odds is a fallacy.\u0026rdquo;\nThis talk has been brewing for over two years. While talking to Patrick Debois at DevOpsDays Minneapolis 2014 the conversation meandered outside the typical confines of dev \u0026amp; ops. It was around the time Patrick joined Small Town Heroes, and I was reflecting on the parallels between themes at DevOpsDays and my work at Critical Mass. It felt like the issues we as a community were discussing were much more easily reconciled than the approach differences of Creative and Technology in my work environment.\nI have come to believe that the premise that dev and ops are at odds is a fallacy. Sure, there are perverse organizational structures that silo expertise and incent them against one another, but I\u0026rsquo;m operating under the assumption that such easily identified wrongs aren\u0026rsquo;t present. What I was left to ponder (and later enumerate) were strategies and tactics that can be used to create high-performance, cohesive, multi-disciplinary teams.\nHere\u0026rsquo;s an abridged outline of what I covered in the talk:\nMission statement Participating \u0026amp; sharing in each others\u0026rsquo; work process Building trust Surprise reduction Psychological safety \u0026ldquo;Understand your assumptions and\nanticipate other\u0026rsquo;s needs.\u0026rdquo;\nDevOps is about tools and culture. As much as I enjoy the occasional tool-smithing indulgence, I find the topic of leadership\u0026rsquo;s role in creating an environment that nurtures high-performance teams even more enjoyable and satisfying. Leave a comment or ping me on Twitter if you\u0026rsquo;d like to discuss further. Cheers.\nhttps://speakerdeck.com/lanyonm/creative-and-technology-a-partnership-devopsdays-msn-2016 ","date":"2016-11-02","permalink":"/speaking/creative-and-technology-a-partnership-devopsdays-msn/","section":"Speaking","summary":"","title":"Creative \u0026 Technology: A Partnership - DevOpsDays MSN 2016"},{"content":"","date":null,"permalink":"/tags/culture/","section":"Tags","summary":"","title":"Culture"},{"content":"","date":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"Devops"},{"content":"","date":null,"permalink":"/tags/devopsdays/","section":"Tags","summary":"","title":"Devopsdays"},{"content":"","date":null,"permalink":"/tags/empathy/","section":"Tags","summary":"","title":"Empathy"},{"content":"","date":null,"permalink":"/tags/leadership/","section":"Tags","summary":"","title":"Leadership"},{"content":"","date":null,"permalink":"/","section":"Notes \u0026 thoughts from Mike Lanyon","summary":"","title":"Notes \u0026 thoughts from Mike Lanyon"},{"content":"","date":null,"permalink":"/speaking/","section":"Speaking","summary":"","title":"Speaking"},{"content":"","date":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"/articles/","section":"Articles","summary":"","title":"Articles"},{"content":"","date":null,"permalink":"/tags/elasticsearch/","section":"Tags","summary":"","title":"Elasticsearch"},{"content":"","date":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java"},{"content":"","date":null,"permalink":"/tags/kibana/","section":"Tags","summary":"","title":"Kibana"},{"content":"While parsing raw log files is a fine way for Logstash to ingest data, there are several other methods to ship the same information to Logstash. These methods each have trade-offs that may make them more or less suitable for your particular situation. I have posted about multiline tomcat log parsing before, and this post is an attempt to compare that and other methods I\u0026rsquo;ve explored: log4j as JSON, log4j over TCP, and raw log4j with the multiline codec.\nThese examples were developed on one machine but are designed to work in an environment where your ELK stack is on a separate machine/instance/container. In the multi-machine environment Filebeat (formerly logstash-forwarder) would be used in cases where the example uses the file input.\nFor posterity\u0026rsquo;s sake, these are the software versions used in this example:\nJava 7u67 Spring 4.2.3 Logstash 2.1.0 Elasticsearch 2.1.1 Kibana 4.3.1 I first began to author this post on Nov, 28th 2014, so please forgive any options presented that are no longer in favor. Also, you\u0026rsquo;ll notice that slf4j is used as an abstraction for log4j in the code samples.\nLog4j As JSON #This method aims to have log4j log as JSON and then use Logstash\u0026rsquo;s file input with a json codec to ingest the data. This will avoid unnecessary grok parsing and the thread unsafe multiline filter . Seeing json-formatted logs can be jarring for a Java dev (no pun intended), but reading individual log files should be a thing of the past once you\u0026rsquo;re up and running with log aggregation. Also, you can run two appenders in parallel if you have the available disk space.\nInstead of using a PatternLayout with a heinously complex ConversionPattern, let\u0026rsquo;s have a look at log4j-jsonevent-layout . The prospect of a solution that is entirely in configuration fits the bill, and I can live with yet another logging dependency.\nIncluding the dependency in a pom.xml is quite easy:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;net.logstash.log4j\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;jsonevent-layout\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.7\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; The log4j.properties will look familiar, and I\u0026rsquo;ll explain UserFields in a bit.\nlog4j.rootLogger=debug,json log4j.appender.json=org.apache.log4j.DailyRollingFileAppender log4j.appender.json.File=target/app.log log4j.appender.json.DatePattern=.yyyy-MM-dd log4j.appender.json.layout=net.logstash.log4j.JSONEventLayoutV1 log4j.appender.json.layout.UserFields=application:playground,environment:dev And the Logstash config is as you\u0026rsquo;d expect:\ninput { file { codec =\u0026gt; json type =\u0026gt; \u0026#34;log4j-json\u0026#34; path =\u0026gt; \u0026#34;/path/to/target/app.log\u0026#34; } } output { stdout {} } The values set in the UserFields are important because they allow the additional log metadata (taxonomy) to be set in the application configuration. This is information about the application and environment that will allow the log aggregation system to categorize the data. Because we\u0026rsquo;re using the file input plugin we could also use add_field, but this would require separate file plugins statements for every application. Certainly possible, but even with configuration management more of a headache than the alternative. Also, the path parameter of the file plugin is an array so we can specify multiple files with ease.\nIf everything is set correctly, log messages should look like this in Kibana:\nA log message from Playground using log4j-jsonevent-layout As you can see, the UserFields are parsed into Logstash fields. If you prefer these values to be set via command line and environment variable, the library provides a way that will override anything set in the log4j.properties.\nLog4j over TCP #This method uses log4j\u0026rsquo;s SocketAppender and Logstash\u0026rsquo;s log4j input . Log events are converted into a binary format via the SocketAppender and streamed to the log4j input. The advantages here are that the new log4j appender can be added without additional dependencies and that we are able to avoid dealing with the multiline filter. Let\u0026rsquo;s look at the implementation before digging into the shortcomings.\nHere\u0026rsquo;s a snippet of the log4j.properties :\nlog4j.rootLogger=debug,tcp log4j.appender.tcp=org.apache.log4j.net.SocketAppender log4j.appender.tcp.Port=3456 log4j.appender.tcp.RemoteHost=localhost log4j.appender.tcp.ReconnectionDelay=10000 log4j.appender.tcp.Application=playground And the corresponding snippet of Logstash config:\ninput { log4j { mode =\u0026gt; \u0026#34;server\u0026#34; host =\u0026gt; \u0026#34;0.0.0.0\u0026#34; port =\u0026gt; 3456 type =\u0026gt; \u0026#34;log4j\u0026#34; } } output { stdout {} } One of the log4j configurations above that you rarely see is Application. When this parameter is set Logstash will parse it into an event field. This is handy, but may not satisfy your logging taxonomy - exposing one of this method\u0026rsquo;s shortcomings: tagging log events with application and environment identifying information. The way this is typically done in the Logstash config is with add_field on an input plugin. Taking the typical approach would mean a different input plugin/port for each java app sending logs - not fun to manage at scale!\nMapped Diagnostic Context #Mapped Diagnostic Context (MDC) provides a way to enrich standard log information via a map of values of interest. Thankfully the log4j plugin will parse MDC hashes into log event fields. The MDC is managed on a per-thread basis, but a child thread automatically inherits a copy of the MDC from it\u0026rsquo;s parent. This means that log taxonomy can be set to the MDC in the application\u0026rsquo;s main thread and affect every log statement for its entire lifespan.\nHere\u0026rsquo;s minimalistic example:\nimport org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; public class LoggingTaxonomy { private static final Logger log = LoggerFactory.getLogger(LoggingTaxonomy.class); static public void main(String[] args) { MDC.put(\u0026#34;environment\u0026#34;, System.getenv(\u0026#34;APP_ENV\u0026#34;)); // run the app log.debug(\u0026#34;the app is running!\u0026#34;); } } With a PatternLayout conversion pattern like %d{ABSOLUTE} %5p %c{1}:%L - %X{environment} - %m%n and APP_ENV set to dev you\u0026rsquo;d expect to see a log statement like:\n15:23:03,698 DEBUG LoggingTaxonomy:34 - dev - the app is running! How and where to integrate the MDC values will vary widely based on the framework used by the application, but every framework I\u0026rsquo;ve ever used has an appropriate place to set this information. In Spring MVC with Java Config it can go in one of the AppInitializer methods. There are additional uses for MDC, which I\u0026rsquo;ll write about in a future post.\nOnce all the code and config is correct, the enhanced logs will flow into your Kibana dashboard like so:\nA log message from Playground using Log4j over TCP and MDC for additional log event fields A few things to note about this approach:\nThe logging level is stored in priority, not level as is with log4j-jsonevent-layout There is no source_host field, so you may need to add that via MDC as well Raw Log4j and the Multiline Codec #My multiline parsing post used the multiline filter plugin, but as mentioned above that plugin isn\u0026rsquo;t threadsafe. I wanted to provide a slight update to that approach that uses the multiline codec instead of the filter. I\u0026rsquo;ve modified the original example as little as possible and integrated the relevant bits into the playground app.\nHere\u0026rsquo;s a snippet of the log4j.properties :\nlog4j.rootLogger=debug,file log4j.appender.file=org.apache.log4j.DailyRollingFileAppender log4j.appender.file.File=target/file.log log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS ZZZ} | %p | %c - %m%n And the corresponding snippet of Logstash config (the grok pattern file can be found here ):\ninput { file { type =\u0026gt; \u0026#34;raw-file\u0026#34; path =\u0026gt; \u0026#34;/path/to/target/file.log\u0026#34; add_field =\u0026gt; { \u0026#34;application\u0026#34; =\u0026gt; \u0026#34;playground\u0026#34; \u0026#34;environment\u0026#34; =\u0026gt; \u0026#34;dev\u0026#34; } codec =\u0026gt; multiline { patterns_dir =\u0026gt; \u0026#34;/path/to/logstash/patterns\u0026#34; pattern =\u0026gt; \u0026#34;(^%{TOMCAT_DATESTAMP_PATTERN})|(^%{CATALINA_DATESTAMP_PATTERN})\u0026#34; negate =\u0026gt; true what =\u0026gt; \u0026#34;previous\u0026#34; } } } output { stdout {} } You\u0026rsquo;ll notice that I used add_field to add application and environment fields because adding those to the ConversionPattern and grok parsers would\u0026rsquo;ve required some heavy lifting. If I\u0026rsquo;d built-out this solution fully, I would have integrated these via MDC as described above and made the ConversionPattern and Grok parse updates.\nThe log message will look like this in Kibana:\nA log message from Playground using the file input and multiline codec to parse a raw Log4j In my opinion there are several shortcomings to this approach:\nCreating multiline parsers can be tough. Grok parse patterns are tightly coupled to Conversion pattern and require adjustments in both places for changes. Developers won\u0026rsquo;t be able to add MDC information and have it automagically show up in the log aggregation system. Other Options #Log4j isn\u0026rsquo;t the only logging solution for Java. Logback is growing in popularity and implements the slf4j API making it swappable with Log4j or JUL. The logstash-logback-encoder looks particularly robust. If you\u0026rsquo;re coming from a Log4j implementation be sure to use the LogstashTcpSocketAppender, not the LogstashSocketAppender. The latter uses UDP and debugging an incident where log messages may have been dropped is a recipe for disaster. Given more time this would be my next exploration.\nSummary #I hope the comparison of these methods is helpful. If it\u0026rsquo;s not already clear, my preference is log4j as JSON using log4j-jsonevent-layout and Filebeat. Logstash-forwarder or Filebeat is already on many of our servers, so this is an easy approach for us. I\u0026rsquo;m interested to hear others\u0026rsquo; experiences managing their log aggregation pipeline.\n","date":"2015-12-29","permalink":"/articles/log-aggregation-log4j-spring-logstash/","section":"Articles","summary":"","title":"Log Aggregation with Log4j, Spring, and Logstash"},{"content":"","date":null,"permalink":"/tags/log4j/","section":"Tags","summary":"","title":"Log4j"},{"content":"","date":null,"permalink":"/tags/logstash/","section":"Tags","summary":"","title":"Logstash"},{"content":"","date":null,"permalink":"/tags/maven/","section":"Tags","summary":"","title":"Maven"},{"content":"","date":null,"permalink":"/tags/monitoring/","section":"Tags","summary":"","title":"Monitoring"},{"content":"","date":null,"permalink":"/tags/operations/","section":"Tags","summary":"","title":"Operations"},{"content":"","date":null,"permalink":"/tags/software/","section":"Tags","summary":"","title":"Software"},{"content":"","date":null,"permalink":"/tags/spring/","section":"Tags","summary":"","title":"Spring"},{"content":"On the heels of the previous post about continuously delivering documentation I wanted to show how easy it is to integrate dependency vulnerability checks and reports into a Maven-based delivery pipeline. The OWASP Top 10 2013 contains an entry about Using Components with Known Vulnerabilities , so being able to check project dependencies against a canonical list of vulnerable libraries is critical to compliance with the Top 10. The OWASP Dependency Check utility uses NIST\u0026rsquo;s National Vulnerability Database (NVD) to identify the vulnerable dependencies, so the list is always up-to-date.\nThe Dependency Check utility is conveniently wrapped by the dependency-check-maven plugin. The usage information contains several examples and the configuration allows plenty of tuning for the analyzer. With a single configuration parameter the plugin can fail a build if a vulnerable dependency is found.\nCVSS Scores and the POM #The Common Vulnerability Scoring System (CVSS) uses several aspects of exploit-ability and impact to determine a base score. Each vulnerability in the NVD has a base score attached, and anything above 4.0 is considered Medium or High severity (CVSS overview ).\nEvents following the publishing of a CVE can affect the CVSS score (e.g., an exploit kit is published). These adjusted scores are referred to as temporal scores, and CVSS provides a calculator to help understand the effect. The calculator is also quite handy to help understand what influences a score.\nThanks to the examples , configuring the POM is quite easy. Here\u0026rsquo;s the configuration I\u0026rsquo;m using in my pom.xml :\n\u0026lt;project\u0026gt; \u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.owasp\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;dependency-check-maven\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.3.3\u0026lt;/version\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;cveValidForHours\u0026gt;12\u0026lt;/cveValidForHours\u0026gt; \u0026lt;failBuildOnCVSS\u0026gt;4\u0026lt;/failBuildOnCVSS\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;check\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; \u0026lt;/project\u0026gt; The configuration above will fail the Maven run if a dependency with a CVSS score above 4 is found and will recheck the NVD only every 12 hours. To run a standalone dependency check:\nmvn dependency-check:check Report Integration #Integrating a dependency check report into the generated Maven site is also quite easy thanks to the examples. Here\u0026rsquo;s the relevant snippet of the pom.xml :\n\u0026lt;configuration\u0026gt; \u0026lt;reportPlugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.owasp\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;dependency-check-maven\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.3.3\u0026lt;/version\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;name\u0026gt;Dependency Check\u0026lt;/name\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;reportSets\u0026gt; \u0026lt;reportSet\u0026gt; \u0026lt;reports\u0026gt; \u0026lt;report\u0026gt;aggregate\u0026lt;/report\u0026gt; \u0026lt;/reports\u0026gt; \u0026lt;/reportSet\u0026gt; \u0026lt;/reportSets\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/reportPlugins\u0026gt; \u0026lt;/configuration\u0026gt; A \u0026ldquo;Dependency Check\u0026rdquo; link will be added to the Project Reports page of the generated site that points to a page that looks like this:\nOWASP's Dependency Check showing CVE-2015-6420 in Apache's commons-collections:3.2.1 For illustrative purposes I included a known vulnerable dependency. Running mvn site didn\u0026rsquo;t cause the build to fail because the plugin configuration used during the site lifecycle phase is separate from what was defined for the standalone check.\nSummary #While not a full solution for security in your continuous delivery process, the dependency-check-maven plugin is a great way for a Java project to check the A9 box on the OWASP Top 10. Additionally, the visibility that the report gives dependency security is fantastic.\nOne handy feature I didn\u0026rsquo;t mention is that a suppression list can be used to ignore positives. This can be used for false positives, or to ignore acknowledged vulnerabilities currently awaiting resolution in the backlog. This would allow the CI process to continue to run while the project is brought into compliance.\n","date":"2015-12-22","permalink":"/articles/continuous-security-owasp-java-vulnerability-check/","section":"Articles","summary":"","title":"Continuous Security with OWASP's Dependency Check Maven Plugin"},{"content":"","date":null,"permalink":"/tags/documentation/","section":"Tags","summary":"","title":"Documentation"},{"content":"","date":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"Github"},{"content":"","date":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security"},{"content":"Part of continuous delivery is continuously delivering documentation along with the software. I\u0026rsquo;d go so far as to argue that it\u0026rsquo;s part of the software. In the past I\u0026rsquo;ve had to remember to run mvn site periodically to ensure that the latest javadoc and dependency updates report get created, but there\u0026rsquo;s gotta be a better way!\nThe key to this integration is having Travis authenticate back to GitHub to publish to gh-pages in a secure way. The default suggestion for publishing with GitHub\u0026rsquo;s site-maven-plugin is to add your username \u0026amp; password or an oauth token to ~/.m2/settings.xml. This won\u0026rsquo;t work for Travis-CI and would leak the oauth token.\nPOM Configuration #The site-maven-plugin configuration is exactly the same as GitHub\u0026rsquo;s example , but that\u0026rsquo;s to be expected. The important configuration is to allow the oauth token to be read from an environment variable (excerpt from pom.xml ):\n\u0026lt;project\u0026gt; \u0026lt;properties\u0026gt; \u0026lt;github.global.server\u0026gt;github\u0026lt;/github.global.server\u0026gt; \u0026lt;github.global.oauth2Token\u0026gt;${env.GITHUB_OAUTH_TOKEN}\u0026lt;/github.global.oauth2Token\u0026gt; \u0026lt;/properties\u0026gt; \u0026lt;/project\u0026gt; To be able to run mvn site locally you\u0026rsquo;ll need to run export GITHUB_OAUTH_TOKEN=\u0026quot;your-github-personal-access-token\u0026quot; or add that line to your dotfiles. To create the token follow these instructions . The token I created has repo and user:email access.\nTravis-CI Configuration #Encrypted Environment Variable #Getting your GitHub token into the Travis-CI environment var is very well documented . Here\u0026rsquo;s a handy copy/paste of the command you\u0026rsquo;ll want to run to encrypt your environment variable:\ntravis encrypt GITHUB_OAUTH_TOKEN=\u0026#34;your-github-personal-access-token\u0026#34; --add env.global The one thing not covered in the Travis-CI docs page is that you can have multiple encrypted environment variables like so:\nenv: global: secure: bigEncryptedString/One secure: bigEncryptedString/Two Build Timeout #Another thing to consider is that the site-maven-plugin can take quite a while to prepare and upload the site html to your gh-pages branch. By default this process does not generate log statements and Travis-CI may time-out. You can either have Travis extend the timeout or enable debug logging for Maven: mvn site -X. I chose the second option because I didn\u0026rsquo;t want some other issue to cause long build times. The .travis.yml has the full details.\nSummary #You can see this working in my Spring playground repository which publishes here . Please let me know if you find this helpful or have suggestions for improvements.\n","date":"2015-12-19","permalink":"/articles/publish-maven-site-github-pages-travis-ci/","section":"Articles","summary":"","title":"Publishing a Maven Site to GitHub Pages with Travis-CI"},{"content":"","date":null,"permalink":"/tags/travis-ci/","section":"Tags","summary":"","title":"Travis-Ci"},{"content":"","date":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws"},{"content":"","date":null,"permalink":"/tags/chatops/","section":"Tags","summary":"","title":"Chatops"},{"content":"You probably searched \u0026ldquo;pingdom alerts in hipchat\u0026rdquo; or \u0026ldquo;pingdom hipchat integration\u0026rdquo; and were unhappy to find that there\u0026rsquo;s no direct method to integrate the two services. I was too - but it gave me the chance to use the AWS API Gateway and AWS Lambda to connect the two services. I assume you\u0026rsquo;re relatively familiar with the functionality that API Gateway and Lambda provide as well as getting-started experience with Node.js.\nPingdom Webhooks #The documentation on the Pingdom site describes how to find the webhook payload structure, so I followed their advice and observed the up \u0026amp; down webhook events. I don\u0026rsquo;t use it this demo, but each of the webhooks a X-Request-Id header was sent with a value like 6645779c-18a9-473d-808e-2b74450c7347. You may find this useful for tracing purposes.\nDown #As you can see, the assign webhook is a GET request with the message payload as an URI encoded json string. It would be nice if this was a POST, but ¯\\_(ツ)_/¯.\nGET /webhook-endpoint?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22assign%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22down%22%7D The decoded querystring looks like:\nmessage = { \u0026#34;check\u0026#34;: \u0026#34;1834565\u0026#34;, \u0026#34;checkname\u0026#34;: \u0026#34;just a test\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;www.example.com\u0026#34;, \u0026#34;action\u0026#34;: \u0026#34;assign\u0026#34;, \u0026#34;incidentid\u0026#34;: 8765, \u0026#34;description\u0026#34;: \u0026#34;down\u0026#34; } Up #Aka notify_of_close or resolved:\nGET /webhook-endpoint?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22notify_of_close%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22up%22%7D message = { \u0026#34;check\u0026#34;: \u0026#34;1834565\u0026#34;, \u0026#34;checkname\u0026#34;: \u0026#34;just a test\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;www.example.com\u0026#34;, \u0026#34;action\u0026#34;: \u0026#34;notify_of_close\u0026#34;, \u0026#34;incidentid\u0026#34;: 8765, \u0026#34;description\u0026#34;: \u0026#34;up\u0026#34; } Testing with curl #For testing purposes you may want a quick curl statement to act as Pingdom (so you don\u0026rsquo;t have to deliberately take a monitored endpoint down):\ncurl -H 'X-Request-Id: 6645779c-18a9-473d-808e-2b74450c7347' https://1x1x1x1x1x.execute-api.us-east-1.amazonaws.com/prod/pingdom-webhook?message=%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22assign%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22down%22%7D Now that we understand Pingdom\u0026rsquo;s webhook a bit better, let\u0026rsquo;s have a look at the AWS parts.\nAWS Lambda #Before we configure the API Gateway, let\u0026rsquo;s have a look at the Lambda function. We do this first because you\u0026rsquo;ll need to select the Lambda when you create the API Gateway resource. I chose to use Node.js, but the code is straightforward and should be able to be ported to python easily.\nWhen creating a Lambda function you\u0026rsquo;ll be prompted to select a blueprint. Any will do, but microservice-http-endpoint will most closely mirror the functionality of our Lambda. With a blueprint selected, you\u0026rsquo;ll need to configure the name, runtime, handler (entry point), IAM role, memory, and timeout for the Lambda. You will likely need to create a basic IAM role to allow your Lambda (the AWS console will help you do this), and you\u0026rsquo;ll want to decrease the memory requirement to 128MB.\nThe entry point for the Lambda in this example is index.pingdomToHipchat, and per the Lambda spec the function takes the event and context objects. The full repo is on GitHub , and I\u0026rsquo;ve included the index.js below:\n\u0026#39;use strict\u0026#39;; var http = require(\u0026#39;https\u0026#39;); var config = require(\u0026#39;./config.js\u0026#39;); exports.pingdomToHipchat = function(event, context) { // the contents of event is dependent on the configuration of API Gateway // console.log(\u0026#39;the message is\u0026#39;, event.message); // there some things that the decodeURI method doesn\u0026#39;t clean up for us var msg = JSON.parse(decodeURI(event.message).replace(/\\+/g, \u0026#39; \u0026#39;).replace(/%3A/g, \u0026#39;:\u0026#39;).replace(/%2C/g, \u0026#39;,\u0026#39;)); console.log(\u0026#39;the message json is:\\n\u0026#39;, msg); var hc_msg = { color: msg.description === \u0026#39;down\u0026#39; ? \u0026#39;red\u0026#39; : \u0026#39;green\u0026#39;, message: msg.checkname + \u0026#39; is \u0026#39; + msg.description + \u0026#39; (\u0026#39; + msg.host + \u0026#39;)\u0026#39;, notify: false, message_format: \u0026#39;text\u0026#39;, }; console.log(\u0026#39;hipchat message:\\n\u0026#39;, hc_msg); var http_opts = { host: \u0026#39;api.hipchat.com\u0026#39;, port: 443, method: \u0026#39;POST\u0026#39;, path: \u0026#39;/v2/room/\u0026#39; + config.hipchat.room + \u0026#39;/notification?auth_token=\u0026#39; + config.hipchat.token, headers: { \u0026#39;Content-Type\u0026#39;: \u0026#39;application/json\u0026#39;, } }; var req = http.request(http_opts, function(res) { res.setEncoding(\u0026#39;utf8\u0026#39;); res.on(\u0026#39;data\u0026#39;, function (chunk) { console.log(\u0026#39;BODY:\u0026#39;, chunk); }); res.on(\u0026#39;end\u0026#39;, function () { if (res.statusCode === 204) { console.log(\u0026#39;success - message delivered to hipchat\u0026#39;); context.succeed(\u0026#39;message delivered to hipchat\u0026#39;); } else { console.log(\u0026#39;failed with\u0026#39;, res.statusCode); context.fail(\u0026#39;hipchat API returned an error\u0026#39;); } }); }); req.on(\u0026#39;error\u0026#39;, function(e) { console.log(\u0026#39;problem with request:\u0026#39;, e.message); context.fail(\u0026#39;failed to deliver message to hipchat\u0026#39;); }); req.write(JSON.stringify(hc_msg)); req.end(); }; On line 11 event.message is decoded and then cleaned further to compensate for the remaining encoding weirdness. Once I figured out how to translate the querystring params in API Gateway, iterating on this was the last bit of magic to get a json representation of the Pingdom alert in the Lambda function. The rest of the code creates the HipChat notification API request \u0026amp; payload, and then sends the request.\nThe console.log statements are sent to CloudWatch - which can help you audit or debug during development. The built-in Lambda test functionality is also captures this output, and is the quickest way to ensure that changes to the Lambda function as expected. This is the test event for the Lambda above:\n{ \u0026#34;requestId\u0026#34;: \u0026#34;6645779c-18a9-473d-808e-2b74450c7347\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;%7B%22check%22%3A%20%221834565%22%2C%20%22checkname%22%3A%20%22just%20a%20test%22%2C%20%22host%22%3A%20%22www.example.com%22%2C%20%22action%22%3A%20%22notify_of_close%22%2C%20%22incidentid%22%3A%208765%2C%20%22description%22%3A%20%22up%22%7D\u0026#34; } Lastly, if you want to be able to copy/paste this code into the AWS Lambda console for testing you\u0026rsquo;ll need to remove the config.js require on line 4 and replace the two values on line 27 with your HipChat token and room id. More on finding these values for your setup below .\nNow that we have the Lambda squared away, let\u0026rsquo;s see how it gets wired up with the API Gateway.\nAWS API Gateway #The API Gateway takes the Pingdom GET request and populates the event object passed to the Lambda. This step would be much easier if the Pingdom webhook used POST instead of GET, but you\u0026rsquo;ll learn something interesting about the API Gateway as a result.\nInside the API Gateway console create a new API, create a resource, and create a GET method. When creating the method, you\u0026rsquo;ll need to select \u0026ldquo;Lambda Function\u0026rdquo; as the integration type the region the Lambda function is deployed into, and the Lambda name (which will auto-complete).\nAt this point you should see something like this:\nThe API Gateway method before configuration. There\u0026rsquo;s a couple things we need to do to translate the incoming Pingdom GET into the event that the Lambda expects. The first is to define the method request. We need to specify the X-Request-Id header and message query string as shown below:\nThe API Gateway method request configuration. The magic happens in the integration request configuration. Select \u0026ldquo;Lambda Function\u0026rdquo; for the integration type, select your Lambda function name, and add an application/json content-type Mapping Template. The previous step made X-Request-Id and message available to be mapped into the event as follows:\nThe API Gateway method integration request configuration. Header parameters and query string parameters are both fetched via the \u0026quot;$input.params('key')\u0026quot; function. Due to the encoding of the Pingdom webhook, we need to urlEncode the message value to avoid a parse exception.\n{ \u0026#34;requestId\u0026#34;: \u0026#34;$input.params(\u0026#39;X-Request-Id\u0026#39;)\u0026#34;, \u0026#34;message\u0026#34; : \u0026#34;$util.urlEncode($input.params(\u0026#39;message\u0026#39;))\u0026#34; } Once all this is configured, you\u0026rsquo;ll need to \u0026ldquo;Deploy API\u0026rdquo;. Stages are used as environments, so you can test API changes as they roll from development to production. If you make changes you\u0026rsquo;ll need to redeploy the API. Assuming your production environment is prod, you\u0026rsquo;ll receive a url like https://1x1x1x1x1x.execute-api.us-east-1.amazonaws.com/prod/pingdom-webhook .\nPrepping HipChat #HipChat\u0026rsquo;s v2 API requires you to create an integration to push notifications to rooms. This is done via the HipChat admin console and creates an authentication token for the room id specified.\nIf you clone the repo , you\u0026rsquo;ll want to cp config.js.sample config.js and add your token and room id to the config:\nmodule.exports = { hipchat: { token: \u0026#39;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\u0026#39;, room: \u0026#39;1111111\u0026#39; } }; You can find more detail about how to package multiple files for upload to a Lambda function in the README .\nPutting it all together #If all goes well, you should be able to issue curl command above and see a notification appear in the configured HipChat room. Use down and up in the description to see red and green highlighted messages:\nWhat you should see in HipChat if everything goes well. In Pingdom you\u0026rsquo;ll need to go into Alerting \u0026gt; Alerting Endpoints and add a webhook contact method to an Alerting Endpoint used by an Alert Policy that is used by the check you\u0026rsquo;d like to see in HipChat. You likely already have an Alert Policy used for your check, so adding an additional endpoint for that policy should be straightforward.\nI hope this works for you, and please let me know if it doesn\u0026rsquo;t! Big thanks to @ripienaar for his post on translating webhooks that got me thinking about this in the first place.\n","date":"2015-11-25","permalink":"/articles/pingdom-hipchat-integration-aws-lambda/","section":"Articles","summary":"","title":"ChatOps: Pingdom Alerts Pushed into HipChat with AWS Lambda and API Gateway"},{"content":"","date":null,"permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda"},{"content":"","date":null,"permalink":"/tags/nodejs/","section":"Tags","summary":"","title":"Nodejs"},{"content":"As we continue toward ChatOps and making our work visible at work, the next phase of maturing our monitoring systems is to create a query-able interface to our visualization system (Grafana) from HipChat. Grafana is a system I\u0026rsquo;ve become quite fond of and helped author the Chef cookbook for. The HTTP API for Grafana has matured, and the time seemed right to create this integration.\nHigh-level Design #Having read about Librato\u0026rsquo;s ChatOps and seen Etsy\u0026rsquo;s nagios-herald , I had a rough idea of the user experience I wanted. With a head full of hindsight bias, here are some of the requirements:\nA user-friendly query interface in chat (no magic numbers, server-specific names, etc.) Images posted should be available in chat without additional authentication Able to utilize our existing Grafana server I was delighted to find that Stephen Yeargin had already written hubot-grafana , a script that did all the heavy lifting for the first requirement. The Grafana docs site also has a how to integrate Hubot with Grafana article. Stephen\u0026rsquo;s hubot script provides for discovery of dashboards, per-panel queries, template variables, and time-range queries. It\u0026rsquo;s really quite fantastic. However, it assumes that S3 will be used to host the images. While that\u0026rsquo;ll work for most folks (and certainly could work for us), I wanted to be able to use our existing Grafana server to house this integration. To achieve this I had to modify grafana.coffee . More on that below.\nThe default configuration provided by the chef-grafana cookbook includes Nginx as a proxy for grafana-server. For work we wrap the community cookbook to configure TLS, LDAP, and Grafana\u0026rsquo;s datasources. It seemed like a natural extension of visualization\u0026rsquo;s responsibility to have a small app on the Grafana node that can fetch/save rendered panel images and then use Nginx to serve those images. I called that small application grafana-images . More on that below as well.\nModifications to hubot-grafana #As mentioned above, I had to modify the hubot-grafana script to provide an alternate image persistence method (alternative to S3). The coffeescript additions are relatively straightforward:\ncustomFetchAndUpload = (msg, title, url, link) -\u0026gt; requestHeaders = { encoding: \u0026#34;utf8\u0026#34;, Authorization: \u0026#34;Bearer #{grafana_api_key}\u0026#34;, Accept: \u0026#34;application/json\u0026#34; } req_opts = { method: \u0026#34;POST\u0026#34;, url: \u0026#34;#{grafana_images_host}/grafana-images\u0026#34;, headers: requestHeaders, json: { imageUrl: url } } # post to grafana-images request req_opts, (err, res, json) -\u0026gt; robot.logger.debug \u0026#34;grafana-images POST: #{req_opts.url}, content-type[#{res.headers[\u0026#39;content-type\u0026#39;]}]\u0026#34; if res.statusCode == 200 sendRobotResponse msg, title, json.pubImg, link else robot.logger.debug res robot.logger.error \u0026#34;Upload Error Code: #{res.statusCode}\u0026#34; msg.send \u0026#34;#{title} - [Access Error] - #{link}\u0026#34; The Grafana API key is provided to the script by an environment variable and the newly added environment variable HUBOT_USE_GRAFANA_IMAGES determines whether or not to use the customFetchAndUpload code-path. The full diff can be found here .\nAs you can see, the /grafana-images uri is hard-coded. That\u0026rsquo;s because the route used by grafana-images is hard-coded. Also, note that the necessary authentication token is passed along with the json payload. In many ways this function is treating grafana-images as a proxy for Grafana.\nAnother addition to note is the help text I added to the hubot script. You can ask the bot \u0026ldquo;graf help\u0026rdquo; and it\u0026rsquo;ll respond with increasingly complex query samples. Yay for user friendliness!\ngrafana-images #Following my experience with http-stats-collector , Golang seemed like a good choice for the small application. It acts as a proxy and therefore expects only two things: a valid API token and json payload containing the full Grafana panel render url. To give more context to what\u0026rsquo;s happening, here\u0026rsquo;s an http call diagram:\nThe http calls required for hubot-grafana using grafana-images numbered by sequence The customFetchAndUpload function described above is call #4. From there grafana-images will fetch (#5 \u0026amp; #6), save, and return a sharable image url via json (#7). Here\u0026rsquo;s a snippet from grafana-images\u0026rsquo; handlers.go :\n// Fetch image var client = http.Client{} req, err := http.NewRequest(\u0026#34;GET\u0026#34;, image.Url, nil) req.Header.Add(\u0026#34;Accept\u0026#34;, \u0026#34;application/json\u0026#34;) req.Header.Add(\u0026#34;Authorization\u0026#34;, token) resp, err := client.Do(req) if err != nil { log.Fatalf(\u0026#34;http.Get -\u0026gt; %v\u0026#34;, err) w.WriteHeader(500) return } data, err := ioutil.ReadAll(resp.Body) if err != nil { log.Fatalf(\u0026#34;ioutil.ReadAll -\u0026gt; %v\u0026#34;, err) w.WriteHeader(500) return } // Save image fileName := fmt.Sprintf(\u0026#34;%x.png\u0026#34;, md5.Sum(data)) resp.Body.Close() err = ioutil.WriteFile(fmt.Sprintf(\u0026#34;%s/%s\u0026#34;, imagePath, fileName), data, 0666) if err != nil { log.Fatalf(\u0026#34;ioutil.WriteFile -\u0026gt; %v\u0026#34;, err) w.WriteHeader(500) return } // Return image location w.Header().Set(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;) fmt.Fprintf(w, \u0026#34;{\\\u0026#34;pubImg\\\u0026#34;:\\\u0026#34;%s/%s\\\u0026#34;}\u0026#34;, imageHost, fileName) There are several variables assumed to be set:\nimage - the requested imageUrl from the json token - the contents of the Authorization header imagePath - a path on disk to store the saved images imageHost - the host used in the building the json response If everything is configured correctly, the Grafana dashboard panel will be saved to disk and the json sent back to hubot-grafana. Further detail can be found on GitHub . I tired to make all the error messages helpful and actionable, but if you find an error condition that isn\u0026rsquo;t well explained, please open a GitHub issue.\nSecurity #You may have noticed that the app very simply downloads whatever is specified at imageUrl and saves it as a png. This can be dangerous given that nothing checks to ensure that the contents are in-fact an image and not an exploit. Take care to only allow specific traffic to make requests of grafana-images. I may add a check via Golang\u0026rsquo;s png package to ensure proper encoding, but it may be quite some time before that happens (pull requests welcome).\nNginx Config #As mentioned above, I used Nginx to proxy grafana-server. I also use it to proxy grafana-images and serve the saved panel images. Here\u0026rsquo;s a sample conf that should would for this purpose:\n# the grafana-server config has been omitted upstream grafana-images { server 127.0.0.1:8080; } server { # excerpt for grafana-images location /grafana-images { proxy_pass http://grafana-images; allow 10.0.0.11; # ip of server running hubot deny all; } location /saved-images { root /opt; } } Note that the imageHost passed to grafana-images is the FQDN plus the location of the saved images. The value used will be dependent on the web server hosting the saved images.\nOther Uses #Because grafana-images exposes its functionality over a simple HTTP API, expanding its purpose should be straightforward. The app expects an \u0026quot;Authorization: Bearer grafana-token-goes-here\u0026quot; header and a json payload:\n{ \u0026#34;imageUrl\u0026#34;: \u0026#34;https://grafana.example.com/render/dashboard-solo/db/sample-dashboard/?panelId=5\u0026amp;width=1000\u0026amp;height=500\u0026amp;from=now-6h\u0026amp;to=now\u0026amp;var-server=test-server\u0026#34; } Sensu Notifications #At work we have incorporated Grafana panel image embedding functionality into our Sensu HipChat handler. We started with the Sensu community HipChat handler and modified the message body heavily for our purposes.\nThe code to add Grafana panel images to Sensu HipChat notifications is roughly:\nif @event[\u0026#39;check\u0026#39;][\u0026#39;graph_image\u0026#39;] # do some work to get the static image from grafana-images uri = URI.parse(\u0026#39;https://grafana.example.com/grafana-images\u0026#39;) http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true request = Net::HTTP::Post.new(\u0026#39;/grafana-images\u0026#39;) request.add_field(\u0026#39;Content-Type\u0026#39;, \u0026#39;application/json;charset=utf-8\u0026#39;) request.add_field(\u0026#39;Authorization\u0026#39;, \u0026#39;Bearer grafana-token-goes-here\u0026#39;) request.body = { \u0026#39;imageUrl\u0026#39; =\u0026gt; \u0026#34;#{@event[\u0026#39;check\u0026#39;][\u0026#39;graph_image\u0026#39;]}\u0026amp;from=now-6h\u0026amp;to=now\u0026#34; }.to_json begin response = http.request(request) public_image = JSON.parse(response.body)[\u0026#39;pubImg\u0026#39;] message \u0026lt;\u0026lt; \u0026#34;\u0026lt;br /\u0026gt;\u0026lt;a href=\\\u0026#34;#{public_image}\\\u0026#34;\u0026gt;\u0026lt;img src=\\\u0026#34;#{public_image}\\\u0026#34; /\u0026gt;\u0026lt;/a\u0026gt;\u0026lt;br /\u0026gt;\u0026#34; rescue StandardError =\u0026gt; e message \u0026lt;\u0026lt; \u0026#34; - [graph_image fetch failed (#{e})]\u0026#34; end end The @event['check']['graph_image'] value is assumed to be a valid dashboard panel render url without the from/to times: https://grafana.example.com/render/dashboard-solo/db/sample-dashboard/?panelId=5\u0026amp;var-server=test-server\u0026amp;width=1000\u0026amp;height=500. The panelId is can be obtained from the UI of the dashboard.\nWe manage our infrastructure with Chef and it creates all the Sensu checks, thus allowing us to programmatically build the checks. We add a graph_image attribute to the check that contains a panel render url associated with the metric(s) that can help provide context to the Sensu notification. Chef can give the FQDN of the Grafana node as well as the values for template attributes, so it all comes together quite cleanly.\nOther Considerations #One thing not handled by grafana-images is saved image retention. You\u0026rsquo;ll need to create a purge policy that works for you. Once I\u0026rsquo;ve figured out how we\u0026rsquo;re going to handle that, I\u0026rsquo;ll add it here. :)\n","date":"2015-09-30","permalink":"/articles/chatops-hubot-grafana-images-hipchat/","section":"Articles","summary":"","title":"ChatOps: Hubot Grafana Images in HipChat"},{"content":"","date":null,"permalink":"/tags/chef/","section":"Tags","summary":"","title":"Chef"},{"content":"","date":null,"permalink":"/tags/data-visualization/","section":"Tags","summary":"","title":"Data Visualization"},{"content":"","date":null,"permalink":"/tags/golang/","section":"Tags","summary":"","title":"Golang"},{"content":"","date":null,"permalink":"/tags/grafana/","section":"Tags","summary":"","title":"Grafana"},{"content":"","date":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx"},{"content":"","date":null,"permalink":"/tags/ruby/","section":"Tags","summary":"","title":"Ruby"},{"content":"","date":null,"permalink":"/tags/sensu/","section":"Tags","summary":"","title":"Sensu"},{"content":" Last week was DevOpsDays Chicago 2015, my second DevOpsDays Chicago as a co-organizer. We learned a lot in our first year, and I feel that we improved substantially this time around. The anecdotal feedback over the past several days has been overwhelmingly positive, and it\u0026rsquo;s bolstering to feel that outpouring of appreciation after putting in so much effort. My experience as an organizer was much improved as well. We avoided bike-shedding as much as possible, more clearly divided responsibilities, and did our best to improve on our shortcomings in 2014.\nThere are most certainly aspects of the event that we can and will continue to improve upon. We have started the post-mortem process and will be publishing both the 2015 and 2014 post-mortem documents. Keep an eye out for that.\nI wasn\u0026rsquo;t planning on typing up my thoughts about the conference, but a post by Carolyn Van Slyck has been on my mind\u0026hellip;\nParticipation # \u0026ldquo;DevOpsDays is a participant\u0026rsquo;s conference\u0026rdquo;\nI\u0026rsquo;m not sure where I picked up the idea, but I recall saying it in our organizer Slack channel during the run-up to the 2014 event. It is not a platitude. It is at the heart of what makes DevOpsDays an amazing conference series. Open Spaces are typically a new experience for first-time participants and sometimes even dreaded due to the apparent chaos of unstructured time, but every anecdote I heard this year about Open Spaces was it being the best part of the conference.\nIt was my honor and responsibility this year to open the conference. It was with incredible pride that I was able to ask everyone to look down at their badge and see that they are a participant at DevOpsDays, to say that the DevOpsDays community values respect, inclusiveness, and diversity, to say that DevOpsDays is a safe space and remind everyone to be cognizant, and finally to say that whether or not they realized it at the time, their contributions to the community would be valued and respected.\nThese statements are backed by our Code Of Conduct . Originally introduced by DevOpsDays Pittsburgh, this code has become the template used by DevOpsDays events. It conveys a strong belief that is strongly held. As a participant I have spoken up when I have felt the code is violated. As an organizer, it was and is a guide for the tone to set.\nI also introduced Open Spaces, and while I\u0026rsquo;ll never hold a candle to Patrick\u0026rsquo;s introduction I tried to extend the tone of respect and inclusiveness into the four principles of Open Spaces:\nWhoever comes is the right people Whatever happens is the only thing that could have Whenever it starts is the right time When it\u0026rsquo;s over, it\u0026rsquo;s over I heard these statements repeated throughout the afternoon as if to confirm that it is indeed okay to leave an Open Space you\u0026rsquo;re no longer engaged with or continue an Open Space that isn\u0026rsquo;t over. Yes, absolutely.\nReflection #Experiences like Carolyn\u0026rsquo;s are what I and my co-organizers had hoped to nurture. A supportive community that isn\u0026rsquo;t just open to dialog, but actually encourages communication and sharing. At the end of DevOpsDays Chicago 2014, I felt like I\u0026rsquo;d submitted a huge PR to critical OSS that I\u0026rsquo;d been using for years - exhausted, but happy. This year, I feel pride of stewardship and for being part of something that is helping make peoples\u0026rsquo; lives better.\nThanks #I\u0026rsquo;d like to thank the DevOpsDays Chicago 2015 participants, speakers, sponsors, volunteers, and support folks. The community that we convene for just a couple days is why I wanted to organize once again this year. I\u0026rsquo;d also like to thank my co-organizers. It wouldn\u0026rsquo;t be nearly as much fun if you weren\u0026rsquo;t all awesome.\nThe DevOpsDays Chicago 2015 Organizing Team ","date":"2015-09-01","permalink":"/articles/a-participants-conference-devopsdays-chicago/","section":"Articles","summary":"","title":"A Participant's Conference - DevOpsDays Chicago 2015"},{"content":"","date":null,"permalink":"/tags/front-end/","section":"Tags","summary":"","title":"Front-End"},{"content":"I initially intended to speak about how we use configuration management to automate our real user measurement (RUM), but as I was putting together my talk I discovered that the real insight was how the same tools can be used to facilitate the front-end developer\u0026rsquo;s relationship with prod.\nI opened the Ignite by speaking about the lack of operational visibility into the end user\u0026rsquo;s experience and how our initial attempt to close this gap was to use the Navigation Timing data. We use a Golang program called http-stats-collector to collect NavTiming data as well as javascript errors and CSP reports. Looking at the data after it was processed by our monitoring pipeline, we saw that the real insight was how javascript errors could tell the story of the end user\u0026rsquo;s experience - something NavTiming data did not do for us.\nI concluded the talk with a sampling of the systems and templates we have in place to facilitate the collection of data. While this understanding of the end user experience is valuable, the data collection and visualization needs to be turn-key for it to be used across our organization.\nhttps://speakerdeck.com/lanyonm/web-performance-monitoring https://youtu.be/ku3O4HnMXrM?t=395 All photo credits including those in the og:metadata go to Bridget Kromhout or DevOpsDays Minneapolis 2015.\nHere\u0026rsquo;s a transcript of the talk:\nHi, my name is Mike Lanyon, and I work at a digital ad agency called Critical Mass. I would like to talk about web performance monitoring. In DevOps we often talk about monitoring of the systems that provide experiences, but not the monitoring of the end user\u0026rsquo;s experience itself. Critical Mass is a digital experience design agency. That means that we put the customer at the center of our process, through strategy, design, technology and analytics. We want to use our influence with clients to make their customers\u0026rsquo; lives better. More specifically than web performance, I\u0026rsquo;d like to talk about a front end developer\u0026rsquo;s relationship to prod. In years past this relationship may have been through an FTP client, but no-matter the technology there was a gap in operational visibility. The front-end developer\u0026rsquo;s preparation for production has gotten quite robust. Run sass, concat \u0026amp; minify js. Maybe there\u0026rsquo;s a CDN, but then it gets a bit fuzzier\u0026hellip; Where is the operational visibility in this? WebPage test is a tool our teams use to quantify the end user\u0026rsquo;s experience. This chart shows the composite Speed Index of one of our web pages. The Index is the integral of the space above the charted lines, which represent the perceived completion of the page load. But still there is no operational visibility. There\u0026rsquo;s no way for a web developer on my team to understand the performance of a user currently visiting our site. There\u0026rsquo;s nothing analogous to the app log with a stack trace detailing the user\u0026rsquo;s crummy experience. RUM - real user monitoring. This is something that began to pop up a few years ago. This is different than synthetic monitoring because you\u0026rsquo;re collecting data from the experience of real users. Navigation Timing API is one of the most well established means to collecting the real user\u0026rsquo;s performance. There are several solutions that will capture this information for you like Google analytics or NewRelic. Golang. I toyed around with Go a couple years ago, and didn\u0026rsquo;t really understand how it could be useful to me at the time - but it\u0026rsquo;s reputation for being fast seemed like a good fit for collecting RUM performance data. I created https-stats-collector. Inspired by a GDSTeam project called event-store that saves content security policy reports. I don\u0026rsquo;t really want to focus much on the Golang code here - only to highlight that it\u0026rsquo;s open source and would love to have some feedback on the project. There are three primary routes offered by the application: nav-timing, js-error, and csp-report. In addition to nav-timing, I\u0026rsquo;ll talk about js-errors, which is fed by a global javascript error and logs the errors experienced by users. As you can see, after nav-timing data is processed by our monitoring pipeline we\u0026rsquo;re able to create pretty, squiggly line graphs. I really enjoy these, and I think they\u0026rsquo;re great, but they don\u0026rsquo;t help tell the end user\u0026rsquo;s story. This is screencap of Kibana showing the errors captured by the js-errors handler and processed by our ELK stack. When one of the senior developers first saw this, he pointed to a cluster of errors experienced by a single user and said, \u0026ldquo;I wonder what the story is there\u0026rdquo;. We have found that while nav-timing data is really cool, javascript error reporting is the game changer. It gave the teams a level of traceability they\u0026rsquo;d never had before and equated to a front-end version of the app log. This capability is really valuable, but it has to be operationalized and become part of the default path or path of least resistance. We use Chef to put all the bits in place, http-stats-collector, statsd, logstash, elasticsearch, influxdb, kibana, grafana, etc. Configuration management also helps us standardize and scale the implementation. Sample implementations that are framework agnostic help the client/product teams adopt the pattern and integrate it into their work. We have gone so far as to make these RUM collection tools part of our standard stack. In this example a team asked for a simple webserver, and they got the webserver - but packed with all the data collection bits pre-installed. If you take anything from this talk, please consider how to help create the connection between your technology teams and your users. Give your teams the tools to make this connection with prod - to form this relationship with the individual end user. Thank you ","date":"2015-07-08","permalink":"/speaking/web-performance-monitoring-devopsdays-minneapolis/","section":"Speaking","summary":"","title":"Web Performance Monitoring - DevOpsDays MSP 2015"},{"content":"","date":null,"permalink":"/tags/webperf/","section":"Tags","summary":"","title":"Webperf"},{"content":"I recently contributed to the overhaul and 2.0 release of the Grafana Chef cookbook . It was a nearly complete rewrite of the 1.x version, and many decisions were made along the way about what should (and should not) be included in the effort. The cookbook is designed to be as flexible as possible via attributes and to provide the user with a functional setup using the defaults. The previous version used Nginx as a web server, and it made sense to proxy the new Grafana with Nginx in the default setup.\nOne of the initially identified tasks was to provide SSL by default within the cookbook, but that proved to be foolish for two reasons: 1) it was well outside the scope of the Grafana cookbook and 2) SSL configs are highly dependent on several diverse factors ranging from web server version to client browser requirements. Creating a default Nginx SSL setup that was flexibly configurable with attributes was duly marked as out of scope, but not without some discussion .\nFor reference, here\u0026rsquo;s the default Nginx template from the Grafana cookbook:\ntemplate \u0026#39;/etc/nginx/sites-available/grafana\u0026#39; do source node[\u0026#39;grafana\u0026#39;][\u0026#39;nginx\u0026#39;][\u0026#39;template\u0026#39;] cookbook node[\u0026#39;grafana\u0026#39;][\u0026#39;nginx\u0026#39;][\u0026#39;template_cookbook\u0026#39;] notifies :reload, \u0026#39;service[nginx]\u0026#39; mode \u0026#39;0644\u0026#39; owner \u0026#39;root\u0026#39; group \u0026#39;root\u0026#39; variables( grafana_port: node[\u0026#39;grafana\u0026#39;][\u0026#39;ini\u0026#39;][\u0026#39;server\u0026#39;][\u0026#39;http_port\u0026#39;] || 3000, server_name: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_hostname\u0026#39;], server_aliases: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_aliases\u0026#39;], listen_address: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_listen\u0026#39;], listen_port: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_port\u0026#39;] ) end Overriding The Nginx Conf Template #When defining an Nginx template with SSL enabled, it\u0026rsquo;s helpful to have additional variables passed to the template so info like cert locations can be more flexibly defined. Using cookbook and source attributes provided by the recipe would allow for the wrapper cookbook to define a new template, but not allow for additional variables to be passed to the template. By taking advantage of Chef\u0026rsquo;s compile phase , we can alter the template['/etc/nginx/sites-available/grafana'] resource to not only use the template of our choosing, but also pass additional attributes to the resource.\nThe resource definition below is from a Grafana cookbook wrapper recipe:\nbegin r = resources(template: \u0026#39;/etc/nginx/sites-available/grafana\u0026#39;) r.cookbook \u0026#39;wrapper-cookbook\u0026#39; r.source \u0026#39;default/grafana/nginx.conf.erb\u0026#39; r.mode 0644 r.variables( grafana_port: node[\u0026#39;grafana\u0026#39;][\u0026#39;ini\u0026#39;][\u0026#39;server\u0026#39;][\u0026#39;http_port\u0026#39;] || 3000, server_name: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_hostname\u0026#39;], server_aliases: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_aliases\u0026#39;], listen_address: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_listen\u0026#39;], listen_port: node[\u0026#39;grafana\u0026#39;][\u0026#39;webserver_port\u0026#39;], ssl_cert_dir: node[\u0026#39;wrapper-cookbook\u0026#39;][\u0026#39;ssl\u0026#39;][\u0026#39;cert_dir\u0026#39;], ssl_cert_name: node[\u0026#39;wrapper-cookbook\u0026#39;][\u0026#39;ssl\u0026#39;][\u0026#39;cert_name\u0026#39;], ssl_key_dir: node[\u0026#39;wrapper-cookbook\u0026#39;][\u0026#39;ssl\u0026#39;][\u0026#39;key_dir\u0026#39;], ssl_key_name: node[\u0026#39;wrapper-cookbook\u0026#39;][\u0026#39;ssl\u0026#39;][\u0026#39;key_name\u0026#39;], ssl_dhparam_name: node[\u0026#39;wrapper-cookbook\u0026#39;][\u0026#39;ssl\u0026#39;][\u0026#39;dhparam\u0026#39;] ) r.notifies :reload, \u0026#39;service[nginx]\u0026#39;, :immediately rescue Chef::Exceptions::ResourceNotFound Chef::Log.warn \u0026#39;could not find template to override!\u0026#39; end The recipe is grafana and the containing cookbook is wrapper-cookbook. When compared to the template within the Grafana cookbook\u0026rsquo;s _nginx.rb , you can see that five additional SSL-related attributes are passed to the template (lines 12-16).\nA Sample Nginx SSL Template #The nginx.conf.erb will be dependent on your SSL requirements, but here\u0026rsquo;s an example:\n# # This file was generated by Chef for \u0026lt;%= node[\u0026#39;fqdn\u0026#39;] %\u0026gt;. # Do not modify this file by hand! # upstream grafana { server 127.0.0.1:\u0026lt;%= @grafana_port %\u0026gt;; } server { listen \u0026lt;%= \u0026#34;#{@listen_address}:\u0026#34; if !@listen_address.nil? \u0026amp;\u0026amp; !@listen_address.empty? %\u0026gt;\u0026lt;%= @listen_port %\u0026gt;; server_name \u0026lt;%= @server_name %\u0026gt; \u0026lt;%= @server_aliases.join(\u0026#34; \u0026#34;) %\u0026gt;; access_log /var/log/nginx/\u0026lt;%= @server_name %\u0026gt;.access.log; ssl on; ssl_certificate \u0026lt;%= @ssl_cert_dir %\u0026gt;/\u0026lt;%= @ssl_cert_name %\u0026gt;; ssl_certificate_key \u0026lt;%= @ssl_key_dir %\u0026gt;/\u0026lt;%= @ssl_key_name %\u0026gt;; ssl_dhparam \u0026lt;%= @ssl_cert_dir %\u0026gt;/\u0026lt;%= @ssl_dhparam_name %\u0026gt;; ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers \u0026#39;AES256+EECDH:AES256+EDH\u0026#39;; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security max-age=31536000; ssl_stapling on; ssl_stapling_verify on; location / { proxy_pass http://grafana; } } Your mileage will most certainly vary. If you\u0026rsquo;re looking for suggestions on what SSL configs to use Mozilla has put together a great TLS configuration generator: https://mozilla.github.io/server-side-tls/ssl-config-generator/ .\nConclusion #You could argue that this is unnecessary for the Grafana cookbook\u0026rsquo;s _nginx.rb given that the recipe is so simple, but it illustrates the power of Chef\u0026rsquo;s compile phase to override the defaults of the cookbooks used.\nIn addition to a default Nginx setup, the cookbook provides LWRPs for creating datasources, dashboards, organizations, and users. It\u0026rsquo;ll be exciting to see how people use and extend it.\nSample Dashboard configured with the Grafana cookbook. ","date":"2015-06-28","permalink":"/articles/grafana-chef-cookbook-nginx-ssl/","section":"Articles","summary":"","title":"Grafana Chef Cookbook Wrapper to Override Nginx SSL Template"},{"content":"","date":null,"permalink":"/tags/creative/","section":"Tags","summary":"","title":"Creative"},{"content":"I had the privilege of giving an Ignite at DevOpsDays New York 2015 this past week. The theme of the event was inclusivity, complexity, and empathy, which really made for a great event.\nMy talk was about some of the successes and failures I\u0026rsquo;ve experienced when trying to bring DevOps values to my organization and explain how empathy, inclusion, and communication are integral to the successes. I discussed a handful of ideas, including surprise reduction, curiosity, and sharing, and briefly touched upon how diversity in teams enables stronger ideas and better innovation. The journey of cultural change will be different from one organization to the next, but I hope to ignite more conversation on this topic by talking about the changes we\u0026rsquo;ve attempted.\nUltimately, by working to understand other disciplines and finding empathetic, inclusive ways to bridge the gaps between \u0026ldquo;us and them,\u0026rdquo; technology can work more collaboratively with other disciplines, and we can create better, more innovative products as a result.\nHere\u0026rsquo;s a transcript of the talk:\nHi, my name is Mike Lanyon, and I work at an ad agency called Critical Mass. We are a digital experience design agency with a relentless focus on the customer. We\u0026rsquo;re comprised of four primary disciplines: Strategy, Design, Technology and Marketing Science, and we work together to design experiences for large brands. We are not a technology company, but as a digital agency, technology is essential to what we deliver. I like to say that software is the delivery mechanism for brand experiences. And our challenge is to have our disciplines work together in a way that delivers the most innovative yet intuitive experiences. Creating a culture where the disciplines work together to innovate is difficult. Creating a Kata that we practice across the disciplines has been exceedingly difficult. I\u0026rsquo;m going to talk about some of the qualities that I think help lead us in the right direction. If you\u0026rsquo;ve been part of the DevOps community for a while you\u0026rsquo;ll be familiar with lessons of Deming, Goldratt, and Shewhart. While I have found their teachings of statistical process control, system of profound knowledge and theory of constraints valuable, they alone are insufficient to unlocking innovation in our organization. When we look at what drives and delivers value in our organization, we find communication, empathy and inclusivity. You may be thinking, \u0026ldquo;yeah, duh.\u0026rdquo; but I assure you that if it was as simple our theme at DevOpsDays New York wouldn\u0026rsquo;t be what it is. One of the ways I look at communication is surprise reduction. People like to know what\u0026rsquo;s going on and what to expect. Secrecy is toxic. We\u0026rsquo;ve all experienced this, whether it\u0026rsquo;s not knowing enough context for an assignment or simply not knowing that a project is underway. It seems intuitive to keep others in the loop, but surprise reduction is more than that. It\u0026rsquo;s about understanding what you already know that others may not, and anticipating their needs. This takes a level of effort that goes beyond common sense, but it\u0026rsquo;s well worth the energy. When everyone is on the same page - it\u0026rsquo;s more likely that you\u0026rsquo;ll get diverse creative contribution from all disciplines. Innovation sprouts from this proactive communication. I have found that curiosity is another key quality for nurturing innovation. When you\u0026rsquo;re curious you want to know how things work, you want to know what motivates people. I believe it to be an innate quality of intrinsically motivated people. Curiosity has driven designers to learn how to use the web inspector. It\u0026rsquo;s driven information architects to understand how javascript and css power animations, and it drove me to learn about operations. These pieces on the screen where created by my cousin, Sarah Walker who happens to live and work in Brooklyn. These pieces were inspired by how technology has interwoven itself into our lives. If there is one thing you take away from this talk, reflect on your curiosity and act on that interest. If you know someone with that talent or knowledge, ask them about it. Start a conversation. Now put yourself on the other side of that interaction. When you\u0026rsquo;re in the position to answer questions or teach, please, please, please take the time to do so. Continuing the conversation will start a relationship that lasts longer than the dialog. Early in my career, I didn\u0026rsquo;t value sharing. I wasn\u0026rsquo;t confident enough to share work-in-progress and documentation wasn\u0026rsquo;t valued until after it was needed. As I matured I came to understand how sharing enables the organization to gain knowledge as a whole. I came to understand much more holistically that the creative process depends on sharing. I still struggle with how to keep the sharing feedback loop going within the creative process, but the Chicago voting mantra applies: do it early and often. Communication, curiosity, and sharing ladder into a what I believe to be a higher order humanistic quality: empathy. Empathy is understanding and sharing in the feelings of another. And empathy is requisite to innovation. Good design is rooted in Empathy. Imagine trying to design a product without understanding the motivation of the consumer - ok, that’s kinda unheard of, but consider the value of understanding and sharing in emotional needs of both your consumers and your team members. At Critical Mass we not only consider how our software will be received by operators for example, but also think empathetically across discipline lines. Contextualizing a decision with an anecdote while not hiding the details helps the other disciplines relate to what we do in technology. Inclusivity is not to be discounted. It too is requisite to innovation. Ensuring your culture is aware of exclusive behavior and discourages it will allow a more diverse set of contributors, which will ultimately lead to greater perspective and thereby innovation. The creative process, supporting structures, and empathetic culture will never be available as a service or written as code. But we can most certainly apply the values that we hold dear in the DevOps community to this challenge and succeed. Thank you. ","date":"2015-04-30","permalink":"/speaking/innovation-creative-company-devopsdays-newyork/","section":"Speaking","summary":"","title":"Innovation in a Creative Company - DevOpsDays NYC 2015"},{"content":"I tried Golang a couple years ago but didn\u0026rsquo;t see an immediate use for it my work. Fast-forward to February 2015 and add an inspiring blog post by the UK GDS, and I was ready to give Golang another go. The GDSTeam\u0026rsquo;s event-store project was a perfectly sized stepping stone for getting back into Golang. I\u0026rsquo;m passionate about WebPerf and therefore the idea of writing a Go program to collect Navigation Timing API data was a natural choice.\nI had collected Navigation Timing API data in the past, but not in a microservice-esque, reusable way. For example, I had previously built a controller and service into a Grails application to ingest Navigation Timing data and forward it along to StatsD. The idea of building a small, reusable service that collects http statistics appealed to me.\nThe Idea #The general goal was to collect general HTTP statistics, with the first candidate being Navigation Timing data. The rough design of the application is to listen on several content routes that are proxied by the same web server as the primary application. It wasn\u0026rsquo;t clear from the Event Store project exactly how GDSTeam deploys their application, but this made sense as a proof of concept to me.\nA basic diagram of how http-stats-collector would be used. http-stats-collector #The application design evolved as I developed, but as of this writing there are two main components: handlers and recorders. The handlers are HttpFuncHandlers that do the very basic parsing and validation of the HTTP routes. These functions are passed an array of recorder interfaces that implement the functions called by each handler. It\u0026rsquo;s all kinda confusing in words, so let\u0026rsquo;s jump into the code .\nThe Handlers #Here\u0026rsquo;s a excerpt from handlers.go :\ntype NavTimingReport struct { Details NavTimingDetails `json:\u0026#34;nav-timing\u0026#34; statName:\u0026#34;navTiming\u0026#34;` Page string `json:\u0026#34;page-uri\u0026#34; statName:\u0026#34;pageUri\u0026#34;` Referer string `statName:\u0026#34;referer\u0026#34;` UserAgent string `statName:\u0026#34;userAgent` } type NavTimingDetails struct { DNS int64 `json:\u0026#34;dns\u0026#34; statName:\u0026#34;dns\u0026#34;` Connect int64 `json:\u0026#34;connect\u0026#34; statName:\u0026#34;connect\u0026#34;` TTFB int64 `json:\u0026#34;ttfb\u0026#34; statName:\u0026#34;ttfb\u0026#34;` BasePage int64 `json:\u0026#34;basePage\u0026#34; statName:\u0026#34;basePage\u0026#34;` FrontEnd int64 `json:\u0026#34;frontEnd\u0026#34; statName:\u0026#34;frontEnd\u0026#34;` } func NavTimingHandler(recorders []Recorder) http.HandlerFunc { return func(w http.ResponseWriter, req *http.Request) { var timing NavTimingReport if req.Method != \u0026#34;POST\u0026#34; { http.Error(w, \u0026#34;Method not allowed\u0026#34;, http.StatusMethodNotAllowed) w.Header().Set(\u0026#34;Allow\u0026#34;, \u0026#34;POST\u0026#34;) return } if err := json.NewDecoder(req.Body).Decode(\u0026amp;timing); err != nil { http.Error(w, \u0026#34;Error parsing JSON\u0026#34;, http.StatusBadRequest) return } // You could consider this a flaw, but we don\u0026#39;t send the stat anywhere // if it can\u0026#39;t go to one of the recorders. for _, recorder := range recorders { if !recorder.validStat(timing.Page) { http.Error(w, \u0026#34;Invalid page-uri passed\u0026#34;, http.StatusNotAcceptable) return } } // for each recorder we\u0026#39;re sending all the NavTimingDetails stats t := reflect.TypeOf(timing.Details) v := reflect.ValueOf(timing.Details) for i := 0; i \u0026lt; v.NumField(); i++ { for _, recorder := range recorders { stat := recorder.cleanURI(timing.Page) + t.Field(i).Tag.Get(\u0026#34;statName\u0026#34;) val := v.Field(i).Int() recorder.pushStat(stat, val) } } } } As you can see, there are two structs, one nested within the other. The NavTimingReport represents the JSON received in the request body (documented in the readme along with a snippet of javascript that could be used to send the information from the browser to the server).\nThe first several lines of NavTimingHandler is boilerplate HTTP method checking and decoding the JSON request body into a struct. I should note that I used the json.NewDecoder() implementation because it was easier to unit test than the ioutil.ReadAll() + json.Unmarshal combination found in the GDSTeam\u0026rsquo;s event-store.\nLine 33 above is where things get interesting. validStat is a method on the Recorder interface which gives the Recorder the ability to reject a stat for any reason. As is documented in the code, there\u0026rsquo;s a deficiency in the design because if a single Recorder rejects the stat, processing is halted for all Recorders and an error returned to the caller.\nThe last thing the handler does is call cleanURI and pushStat. As before these methods are provided by the Recorder, but I\u0026rsquo;m unhappy with this implementation because of the way it exposes more than necessary to the handler. More on that below . The interesting part of these loops is how reflection is used to pull certain statNames and values from the NavTimingDetails. Additionally, in a few cases the JSON field is named (slightly) differently than how I\u0026rsquo;d like to name the stats that are pushed to the Recorder:\nPage string `json:\u0026#34;page-uri\u0026#34; statName:\u0026#34;pageUri\u0026#34;` This allows the separation the field name used within the JSON in the HTTP body from the stat name used by pushStat. Pretty nifty.\nThe Recorders #Recorder is an interface that provides three methods:\ntype Recorder interface { pushStat(stat string, value int64) bool cleanURI(input string) string validStat(stat string) bool } The method names are hopefully pretty self-explanatory, but there\u0026rsquo;s documentation in each implementation to give more detail. This interface is based on the needs of the StatsD client, but I\u0026rsquo;m sure it could accommodate other statistic processing or storage APIs.\nIt may be easier to just read the full recorders.go given the brevity of the implementation, but I\u0026rsquo;ll review the detail of each implementation for the StatsDRecorder below.\npushStat #The StatsD implementation of pushStat simply wraps cactus \u0026rsquo;s StatsD client. This works with the v1.0.0 version of the dependency.\n// Push stats to StatsD. // This assumes that the data being written is always timing data and we are // always collecting all the samples. func (statsd StatsDRecorder) pushStat(stat string, value int64) bool { if statsd.Statter != nil { err := statsd.Statter.Timing(stat, value, 1.0) if err != nil { log.Fatal(\u0026#34;there was an error sending the statsd timing\u0026#34;, err) return false } } return true } The only two things to note here are that the code above takes care of using a Timer and does not downsample (the \u0026ldquo;1.0\u0026rdquo; arguement in Timing()).\ncleanURI #This is where the code-smell is coming from. For StatsD, I combine the page-uri sent in the JSON with the type of stat: dns, connect, ttfb, basePage, or frontEnd. To differentiate from a page ending in / and other pages, I append index to URIs ending in /, as well as take care of stripping away any unwanted file extensions. I\u0026rsquo;m not convinced this method would survive refactoring to add a second Recorder, but it does a decent job for StatsD. You can see the implementation here , and I\u0026rsquo;ll cover testing below.\nvalidStat #In past Navigation Timing API collection efforts, the team learned that it\u0026rsquo;s important for a web stats collection agent to be able to make decisions about what is valid data and what is not. The endpoint will be open to the internet after all\u0026hellip;\n// The valid page-uri checker for StatsD. We don\u0026#39;t want to accept anything // that the storage would have trouble handing. func (statsd StatsDRecorder) validStat(stat string) bool { return !strings.ContainsAny(stat, \u0026#34;\u0026amp;#\u0026#34;) \u0026amp;\u0026amp; strings.Index(stat, \u0026#34;//\u0026#34;) == -1 } validStat currently does some very rudimentary make-sure-there\u0026rsquo;s-no-query-string checking, but it could be extended to read a file of whitelisted urls, etc.\nMain.go #In a way, this is where the magic happens - but it\u0026rsquo;s also kinda boring. The list of Recorders is constructed and the Handler mounted to its URI.\nvar client statsd.Statter client, err := statsd.NewClient(*statsHostPort, *statsPrefix) if err != nil { log.Fatal(err) } defer client.Close() recorders := []Recorder{StatsDRecorder{client}} http.HandleFunc(\u0026#34;/nav-timing\u0026#34;, NavTimingHandler(recorders)) Flags #Let me take a bit of a detour to gush about how painless flags are to work with in Golang. If you\u0026rsquo;ve parsed input or created usage documentation for a commandline utility before, you know how painful and verbose it can be. Go makes this amazingly simple.\nvar ( port = flag.Int(\u0026#34;port\u0026#34;, 8080, \u0026#34;Server listen port.\u0026#34;) statsHostPort = flag.String(\u0026#34;statsHostPort\u0026#34;, \u0026#34;127.0.0.1:8125\u0026#34;, \u0026#34;host:port of statsd server\u0026#34;) statsPrefix = flag.String(\u0026#34;statsPrefix\u0026#34;, \u0026#34;http-stats-collector\u0026#34;, \u0026#34;the prefix used when stats are sent to statsd\u0026#34;) ) flag.Parse() Not only does this 6 line block take care of parsing the flags, but it generates the --help usage documentation for the executable and handles type checking. It\u0026rsquo;s pretty awesome coming from other languages where the arg parsing can overshadow the actual program.\nTesting #If you go to the GitHub repo for this project you\u0026rsquo;ll see: . It easily took longer to gain this test coverage than it did to write the base program, but good tests are essential for open source projects and help me better understand the code I\u0026rsquo;ve written.\nWhen starting with a new language or framework I have found it is wise to search for and consume what information you can about what works. I found some references to rspec-like frameworks, but SoundCloud found that Go\u0026rsquo;s built-in testing package was sufficient for their needs. I was persuaded by a SoundCloud talk to stay away from any sort of testing framework. This gave me the confidence I needed to proceed without including a testing framework.\nThe Unit Tests #The handlers_test.go is relatively easy to read and understand what\u0026rsquo;s going on. I\u0026rsquo;m sure I could DRY out the code a bit. The basic test structure is:\nfunc TestNavTimingHandlerSuccess(t *testing.T) { req, _ := http.NewRequest(\u0026#34;POST\u0026#34;, \u0026#34;/r\u0026#34;, bytes.NewBufferString(timing_data)) req.Header.Add(\u0026#34;X-Real-Ip\u0026#34;, \u0026#34;192.168.0.1\u0026#34;) req.Header.Add(\u0026#34;User-Agent\u0026#34;, \u0026#34;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0\u0026#34;) resp := httptest.NewRecorder() NavTimingHandler(recorders)(resp, req) const expected_response_code = 200 if code := resp.Code; code != expected_response_code { t.Errorf(\u0026#34;received %v response code, expected %v\u0026#34;, code, expected_response_code) } } The request is set up, response recorder constructed, and the Handler invoked. In this case the response code is compared to what\u0026rsquo;s expect and and error raised if that\u0026rsquo;s not the case.\nThe recorders_test.go is less elaborate because it\u0026rsquo;s just checking that validStat and cleanURI does what they\u0026rsquo;re supposed to.\nIntegration Debugging #I didn\u0026rsquo;t write an integration test per-se, but I did find StatsD dumpMessage mode to be hugely helpful. One might incorrectly assume that debug mode will print the received stats to the console, but some more careful scouring of the internet (and reading the documentation) will lead you to:\ndumpMessages: true This allowed me to see that messages were making it to StatsD as I\u0026rsquo;d expected. Do not enable this in production.\nCode Coverage #In addition to robust testing capabilities, Go ships with built-in code coverage. There are some nuances about how it calculates coverage that you\u0026rsquo;ll want to read , but here\u0026rsquo;s what I ended up using:\ngo test -race -covermode=count -coverprofile=coverage.out \u0026amp;\u0026amp; go tool cover -html=coverage.out The command will calculate coverage and then convert the result into an html report opened in your default browser. Again, pretty nifty.\nMiscellaneous Tips #Makefile #I hadn\u0026rsquo;t written a Makefile in quite a while, but it was a natural way to chain commands together and ensure that tests were run every time the program was compiled, etc. Even if I became more comfortable with Go I think I\u0026rsquo;d continue using a Makefile because of how easy I can clean, test, build, and run. The full makefile is pretty self-explanatory.\nGo Dependency Management - Status Pending #Golang\u0026rsquo;s dependency management is deficient. There\u0026rsquo;s lots of ranting on the internet about this if you\u0026rsquo;re interested in reading more, but I experienced this firsthand when I tried to clone and run on a different machine - and I got a compile error. The issue was that a breaking change was released for the StatsD client between the time I ran go get on the first machine and when I ran go get on the second machine. It is disappointing that Go doesn\u0026rsquo;t provide a built-in mechanism for handling this. There are good userland solutions out there, but it\u0026rsquo;s my understanding that Go 1.5 will have a solution, so I\u0026rsquo;m living dangerously in the meantime.\nGo with Sublime Text #Syntax highlighting has become table-stakes, and the GoSublime package for ST2 takes it to another level. Most notably, GoSublime will run go fmt on every save, ensuring that your code is compliant with the default Go code formatting. I\u0026rsquo;ve seen several open source projects that have git hooks that perform the same function, so it seems quite common.\nGo Interfaces #Before introducing the Recorder interface, I didn\u0026rsquo;t have very good separation of concerns. Jordan Orelli\u0026rsquo;s post on using Interfaces in Go was instrumental in getting myself up to speed.\nDocumentation and package main #The package main is intended to be used for executables and thus the documentation generation is slightly different than library packages. You can use godoc -http=:8080 to generate documentation and view it in a browser, but if you\u0026rsquo;re writing an executable don\u0026rsquo;t expect to see most of your in-code documentation.\nConclusion #Overall, this was an awesome side project. It didn\u0026rsquo;t take too long to complete and will be put to use as part of a stats collection pipeline at work. If you\u0026rsquo;ve gotten this far, I hope this post has helped spark your interest in embarking on a Go project or has answered a nagging question for you.\n","date":"2015-03-29","permalink":"/articles/golang-http-stats-collector/","section":"Articles","summary":"","title":"A Go Program to Collect HTTP Stats"},{"content":"","date":null,"permalink":"/tags/statsd/","section":"Tags","summary":"","title":"Statsd"},{"content":"","date":null,"permalink":"/tags/gpio/","section":"Tags","summary":"","title":"GPIO"},{"content":"","date":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python"},{"content":"","date":null,"permalink":"/tags/raspberry-pi/","section":"Tags","summary":"","title":"Raspberry Pi"},{"content":"Raspberry Pi Init Script #Have you written something handy on your Raspberry Pi and want it to run when the Pi boots up? Making this happen with the Raspbian init system is more difficult than it should be, especially if you want your program to exit correctly and log stdout to a file of your choosing.\nThis example init script uses the SysVinit system currently utilized by the Raspbian operating system and controls a Python program that runs as root (sudo required because GPIO pins are used). The init script takes care of starting the program when the Pi starts, gracefully stopping when told to, and logging to a file without buffering . All this with only a few additional lines of Python to handle the TERM signal when told to stop.\nPlease note: One of the design goals of this init script was to daemonize the Python program with as few modifications to the program as possible. If you are designing a program for distribution, you\u0026rsquo;ll want the program to handle stdout and stderr with proper logging instead of redirection within the init script.\nThe SysVInit Script #This example script is from my Ship-It project. Let\u0026rsquo;s have a look at the entire script before dissecting it:\n#!/bin/sh # ## init script for ship-it # #### BEGIN INIT INFO ## Provides: ship-it ## Required-Start: $remote_fs $syslog $network ## Required-Stop: $remote_fs $syslog $network ## Default-Start: 2 3 4 5 ## Default-Stop: 0 1 6 ## Short-Description: init script for the ship-it box ## Description: We\u0026#39;ll have to fill this out later... #### END INIT INFO PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NAME=ship-it DAEMON=/home/pi/ship-it/main.py DAEMONARGS=\u0026#34;\u0026#34; PIDFILE=/var/run/$NAME.pid LOGFILE=/var/log/$NAME.log . /lib/lsb/init-functions test -f $DAEMON || exit 0 case \u0026#34;$1\u0026#34; in start) start-stop-daemon --start --background \\ --pidfile $PIDFILE --make-pidfile --startas /bin/bash \\ -- -c \u0026#34;exec stdbuf -oL -eL $DAEMON $DAEMONARGS \u0026gt; $LOGFILE 2\u0026gt;\u0026amp;1\u0026#34; log_end_msg $? ;; stop) start-stop-daemon --stop --pidfile $PIDFILE log_end_msg $? rm -f $PIDFILE ;; restart) $0 stop $0 start ;; status) start-stop-daemon --status --pidfile $PIDFILE log_end_msg $? ;; *) echo \u0026#34;Usage: $0 {start|stop|restart|status}\u0026#34; exit 2 ;; esac exit 0 The comments on lines 6-14 are used by SysVinit to determine when during the boot process the script should be called (they\u0026rsquo;re not just comments). The script uses start-stop-daemon , a handy daemon control utility available on Debian and therefore Raspbian systems. There are several key pieces:\n--pidfile $PIDFILE tells start-stop-daemon to use a pidfile (the process id of the daemon) to determine what action to take. This is key to the usefulness of start-stop-daemon because it can read the pidfile and check for the existence of the process. If you\u0026rsquo;ve just asked start-stop-daemon to start an instance of the program but one already exists, start-stop-daemon will tell you \u0026ldquo;process already running\u0026rdquo;. If you ask the start-stop-daemon to stop, it will send a TERM signal to the process id in the pidfile. --make-pidfile tells start-stop-daemon to create a pidfile if one hasn\u0026rsquo;t already been created. Unbuffered logging. This is achieved by using stdbuf from the exec\u0026rsquo;d bash shell and telling it to use line buffer mode. This means that instead of waiting for a certain size of log information to be generated before saving it to the log file, each line will be written to the file as the program produces it. Pidfile cleanup. Line 37 removes the pidfile after the daemon exits. If your Python program doesn\u0026rsquo;t exit when the TERM signal is sent, this file will still be removed - so be sure that your program terminates correctly when it receives the TERM signal. More on this below. Why all this complexity? For the output redirection and unbuffered logging. Without lots of logging code within the Python program, things get messy when you try to daemonize a process. The use of --startas /bin/bash allows the redirection of stdout and stderr to a file via the exec\u0026rsquo;d bash process. Because we have a Python program exec\u0026rsquo;d from bash, bash buffers the output before writing to the log. Using stdbuf allows us to set the buffing configuration of the process. It feels messy, but allows a simple program to function as expected.\nFlexibility #The bash variables on lines 17-21 should provide the flexibility to reuse the init script without any edits to the logic of the script. It would be possible to reuse this script without changing anything below line 20.\nPython to Handle the TERM Signal #In order for the Python program to exit gracefully when the TERM signal is received, it must have a function that exits the program when signal.SIGTERM is received. The function is assigned to a signal handler as seen here:\ndef sigterm_handler(_signo, _stack_frame): \u0026#34;When sysvinit sends the TERM signal, cleanup before exiting.\u0026#34; print(\u0026#34;[\u0026#34; + get_now() + \u0026#34;] received signal {}, exiting...\u0026#34;.format(_signo)) cleanup_pins() sys.exit(0) signal.signal(signal.SIGTERM, sigterm_handler) As mentioned above, this Python program uses the GPIO pins. These pins must be cleaned up so the next program instance can initialize them cleanly. Catching the TERM signal allows GPIO.cleanup() to be called (and a message logged) before the program exits. If you view the full program you\u0026rsquo;ll see that the main loop will also catch a KeyboardInterrupt and cleanup the GPIO pins before exiting.\nInstallation #SysVinit scripts go into the /etc/init.d folder and are linked to from the /etc/rc* directories. The numbered directories represent different runlevels (also seen on lines 10-11 of the init script). Assuming the script is called ship-it.sh and it\u0026rsquo;s currently in the pi user\u0026rsquo;s home directly, here\u0026rsquo;s the installation process:\n$ sudo cp /home/pi/ship-it.sh /etc/init.d/ship-it $ sudo chmod +x /etc/init.d/ship-it $ sudo update-rc.d ship-it defaults Usage #With our SysVinit script installed, we are able to use the service command to interact with it. Use sudo service ship-it status to see the status of the daemon. To start the program: sudo service ship-it start. To stop the program: sudo service ship-it stop.\nIf you attempt to start the program after an instance has already been started, you\u0026rsquo;ll see something like the following:\npi@pecan-pi ~ $ sudo service ship-it start . ok pi@pecan-pi ~ $ sudo service ship-it start process already running. failed! This is referred to as idempotence, a term borrowed from mathematics. In computer science it means that repeating an operation will not change or duplicate the result beyond the effect of the initial operation. This is critically important for this init script because you wouldn\u0026rsquo;t want multiple instances of the Python program running at once.\nOther Considerations #If the daemon is running for a long time, it is possible that the log file size will become untenable. There are several methods to handle this, but I\u0026rsquo;d recommend logrotate . If you choose to add logrotation, you\u0026rsquo;ll want to also change the stdout and stderr redirection to appending (\u0026gt;\u0026gt;) instead of overwriting (\u0026gt;) on line 31 of the init script.\nHappy hacking!\n","date":"2015-01-11","permalink":"/articles/raspberry-pi-init-script-python/","section":"Articles","summary":"","title":"Raspberry Pi Init Script for a Python Program"},{"content":"","date":null,"permalink":"/tags/raspbian/","section":"Tags","summary":"","title":"Raspbian"},{"content":"When I created this blog a couple years ago I only very slightly modified the default Jekyll theme to provide a couple mobile optimizations, but it was always a distant intention to make the site a bit easier on the eyes. Within the Jekyll framework, my desire has always been to:\nhave a responsive site use as little javascript as possible stay compatible with GitHub Pages hosting I didn\u0026rsquo;t make the blog theme a priority (something that was super obvious if you saw the old theme). Earlier this year a friend forwarded Lanyon to me - entirely because of the name. It wasn\u0026rsquo;t until this past weekend that I realized Lanyon met my theme requirements perfectly.\nThe transition process was painless. I had to make sure a few of the customized styles I\u0026rsquo;d added flowed well with the new width, but that was a small hurdle. I\u0026rsquo;d estimate it was much easier to migrate to the new theme than it was to create the blog with all its metadata.\nThanks to @mdo for the open source theme and time spent to make it available.\n","date":"2014-11-26","permalink":"/articles/adopting-the-lanyon-theme/","section":"Articles","summary":"","title":"A New Blog Theme: Lanyon"},{"content":"","date":null,"permalink":"/tags/jekyll/","section":"Tags","summary":"","title":"Jekyll"},{"content":"","date":null,"permalink":"/tags/liquid/","section":"Tags","summary":"","title":"Liquid"},{"content":"","date":null,"permalink":"/tags/development/","section":"Tags","summary":"","title":"Development"},{"content":"","date":null,"permalink":"/tags/sitemesh/","section":"Tags","summary":"","title":"SiteMesh"},{"content":"If you\u0026rsquo;re creating a web application with Spring MVC you\u0026rsquo;ll want to use a view-layer framework. I\u0026rsquo;ve used Grails for several projects at work, and the decorator pattern applied to view-layer files has been a nice way to approach the view-layer architecture. Grails uses SiteMesh under the hood, so I wanted to understand how SiteMesh comes together with Spring.\nPlease note: The examples for this post use Spring 4.0.6 and SiteMesh 2.4.2 (and work with Spring 4.2.x). A new major version of SiteMesh has been released but has not been tested with the suggestions in this post.\nConfiguration #If you\u0026rsquo;ve seen any of my other posts, you\u0026rsquo;ll know that I like Java Config for Spring. The first place SiteMesh is added to Spring MVC in a Java Config project is in the dispatcher servlet filter chains, which takes us to the AppInitializer :\npublic class AppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer { ... @Override protected Filter[] getServletFilters() { CharacterEncodingFilter characterEncodingFilter = new CharacterEncodingFilter(); characterEncodingFilter.setEncoding(\u0026#34;UTF-8\u0026#34;); return new Filter[]{ characterEncodingFilter, new SiteMeshFilter() }; } } I\u0026rsquo;m extending AbstractAnnotationConfigDispatcherServletInitializer , which gives the ability to provide a list of list of Filters via getServletFilters . You can see that I chose to specify a charset filter in addition to the SiteMesh filter.\nThe second piece of configuration is an xml file that tells SiteMesh where to look for templates and how they apply to view-layer files (like jsps). Here\u0026rsquo;s the decorators.xml :\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;decorators defaultdir=\u0026#34;/WEB-INF/decorators/\u0026#34;\u0026gt; \u0026lt;excludes\u0026gt; \u0026lt;pattern\u0026gt;/users\u0026lt;/pattern\u0026gt; \u0026lt;/excludes\u0026gt; \u0026lt;decorator name=\u0026#34;default\u0026#34; page=\u0026#34;default.jsp\u0026#34;\u0026gt; \u0026lt;pattern\u0026gt;*\u0026lt;/pattern\u0026gt; \u0026lt;/decorator\u0026gt; \u0026lt;/decorators\u0026gt; The configuration on line 2 shows that all decorator files are in the WEB-INF/decorators folder. Given that I have overridden Spring\u0026rsquo;s view resolver to look for view files in WEB-INF/views, this makes sense. In lines 3-5 I ignore the /users uri pattern, and on lines 6-8 I setup a decorator.\nUsage #Going forward, I\u0026rsquo;ll be using the term template and decorator interchangeably, and as you\u0026rsquo;ll see below, the term template is quite applicable. I\u0026rsquo;m using Bootstrap for the UI of the playground demo, so there\u0026rsquo;s quite a bit of boilerplate code required on each page. Using the default template I can ensure each page gets exactly the same header, navigation and footer.\nSiteMesh Tags #There are a couple important SiteMesh tags to understanding how the index page gets decorated with default.jsp. The default decorator reads like the shell of an html page:\n\u0026lt;%@ taglib prefix=\u0026#34;dec\u0026#34; uri=\u0026#34;http://www.opensymphony.com/sitemesh/decorator\u0026#34; %\u0026gt; \u0026lt;!doctype html\u0026gt; \u0026lt;html class=\u0026#34;no-js\u0026#34; lang=\u0026#34;en\u0026#34;\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34;\u0026gt; \u0026lt;meta http-equiv=\u0026#34;X-UA-Compatible\u0026#34; content=\u0026#34;IE=edge\u0026#34;\u0026gt; \u0026lt;title\u0026gt;\u0026lt;dec:title default=\u0026#34;playground\u0026#34; /\u0026gt;\u0026lt;/title\u0026gt; global css goes here... \u0026lt;dec:head /\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;div class=\u0026#34;container\u0026#34;\u0026gt; \u0026lt;dec:body /\u0026gt; \u0026lt;/div\u0026gt; global javascript goes here... \u0026lt;/body\u0026gt; Here are the important SiteMesh tags used above:\n\u0026lt;dec:title default=\u0026quot;...\u0026quot; /\u0026gt; - This sets a default title for all pages decorated by this template. If you specify a \u0026lt;title\u0026gt; tag in the head of a view-layer file, it will replace the value set here. \u0026lt;dec:head /\u0026gt; - This is where page-specific \u0026lt;head\u0026gt; elements are added to the head elements provided in the template. It is important to remember that what\u0026rsquo;s in the individual page is added to the \u0026lt;head\u0026gt; of the template. \u0026lt;dec:body /\u0026gt; - This tag is where the \u0026lt;body\u0026gt; content of the view-layer file will be inserted into the template. All of the elements within the \u0026lt;body\u0026gt; of the individual page will be added, so it is important to carefully reason about the nesting of content within the template. There\u0026rsquo;s not a single great example of how these tags are used within SiteMesh, but there\u0026rsquo;s a decent 10-minute starter guide . In addition to the SiteMesh tags, another implementation detail to keep in mind is that that jstl tags are limited to the context of the file in which they are included. You cannot stack all your jstl declarations at the top of the template and have them accessible to the individual page as it renders.\nConclusion #There are other, more fully featured view-layer frameworks, but for my taste the SiteMesh decorator pattern strikes the right balance. By no means is this example a comprehensive demonstration of SiteMesh\u0026rsquo;s capability, but it is a starting point and has hopefully peaked your interest.\n","date":"2014-11-23","permalink":"/articles/spring-4-sitemesh-java-config/","section":"Articles","summary":"","title":"Spring 4 and SiteMesh with Java Config"},{"content":"","date":null,"permalink":"/tags/confluence/","section":"Tags","summary":"","title":"Confluence"},{"content":"","date":null,"permalink":"/tags/markdown/","section":"Tags","summary":"","title":"Markdown"},{"content":"TL;DR #md2confl is a simple Ruby script to automate the conversion of markdown files to Confluence storage format and then upload them to a Confluence server. It uses markdown2confluence to get the markdown to Confluence wiki format and then confluence-soap to convert wiki format to storage format and upload to Confluence. It can be integrated into your CI workflow to help keep your Confluence documentation up to date.\nThe Trouble with Docs\u0026hellip; #Remembering to write documentation can be difficult. Getting others to write it can be even more difficult. I\u0026rsquo;ve found that in-the-code documentation is the easiest to remember to write and in-repo markdown is a close second. Since each programming language and framework has its preferred way to synthesize in-code documentation, my focus here is on the more popular markdown format. There are several ways to render markdown as HTML, but what about putting it in a shared place that\u0026rsquo;s pervasively recognized as the place to go for documentation? Unless you\u0026rsquo;re lucky enough to have something like GitHub or GitLab in your organization, the markdown documentation can be hard to discover and hard to read.\nIn my organization we use Confluence to (sometimes) author and share documentation. Because Confluence is quickly becoming the place we can send someone to learn about one of our products, keeping that documentation fresh is important.\nmd2confl #I pieced together md2confl to automate the conversion and upload of markdown files to Confluence. It uses markdown2confluence to get the markdown to Confluence wiki format and then confluence-soap to convert wiki format to storage format and upload to Confluence. Here\u0026rsquo;s a look at the usage info:\nUsage: md2confl.rb [options...] -s \u0026lt;SPACE_NAME\u0026gt; -i \u0026lt;PAGE_ID\u0026gt; assumes defaults that can be set in options parsing... -i, --pageId PAGE_ID REQUIRED. The Confluence page id to upload the converted markdown to. -s, --space SPACE_NAME REQUIRED. The Confluence space name in which the page resides. -f, --markdownFile FILE Path to the Markdown file to convert and upload. Defaults to 'README.md' -c, --server CONFLUENCE_SERVER The Confluence server to upload to. Defaults to 'http://confluence.example.com' -u, --user USER The Confluence user. Can also be specified by the 'CONFLUENCE_USER' environment variable. -p, --password PASSWORD The Confluence user's password. Can also be specified by the 'CONFLUENCE_PASSWORD' environment variable. -v, --verbose Output more information -h, --help Display this screen The script assumes that a page has already been created for the markdown to be uploaded to. I haven\u0026rsquo;t looked into automating the initial creation of the page, but that would require some additional knowledge, like what the parent page should be, etc. I also acknowledge that this could be more tightly bundled into a standalone executable. Pull requests are welcome!\nThe Plumbing #There is quite a bit of arg parsing for the options discussed above, so I\u0026rsquo;ve cut to the interesting bits. On line 4 and 5, the script grabs the page that the markdown will be uploaded to. On line 14 and line 21 the conversion from markdown to wiki format and then to storage format is performed.\nopts = options[:verbose] ? {} : {log: false} cs = ConfluenceSoap.new(\u0026#34;#{options[:server]}/rpc/soap-axis/confluenceservice-v2?wsdl\u0026#34;, user, password, opts) pages = cs.get_pages(options[:spaceName]) uploader_page = pages.detect { |page| page.id == options[:pageId] } if uploader_page.nil? puts \u0026#34;exiting... could not find pageId: #{options[:pageId]}\u0026#34; exit end begin text = File.read(options[:markdownFile]) @convertedText = \u0026#34;#{Kramdown::Document.new(text).to_confluence}\u0026#34; rescue Exception =\u0026gt; ex warn \u0026#34;There was an error running the converter: \\n#{ex}\u0026#34; end @convertedText = \u0026#34;#{@convertedText}\\n\\n(rendered at #{Time.now.getutc} by md2confl)\u0026#34; uploader_page.content = cs.convert_wiki_to_storage_format(@convertedText) options = {minorEdit: true, versionComment: \u0026#39;updated by md2confl\u0026#39;} cs.update_page(uploader_page) I had to contribute a couple changes to the confluence-soap gem to make all this possible. It was my first time contributing to a Ruby project and I must say that I was impressed by the ease with which I was able to contribute.\n","date":"2014-08-16","permalink":"/articles/markdown-to-confluence-uploader/","section":"Articles","summary":"","title":"Markdown to Confluence Converter \u0026 Uploader"},{"content":"","date":null,"permalink":"/tags/h2/","section":"Tags","summary":"","title":"H2"},{"content":"","date":null,"permalink":"/tags/mybatis/","section":"Tags","summary":"","title":"MyBatis"},{"content":"TL;DR #With the Java Config enhancements in Spring 4, you no longer need xml to configure MyBatis for your Spring application. Using the @MapperScan annotation provided by the mybatis-spring library, you can perform a package-level scan for MyBatis domain mappers. When combined with Servlet 3+, you can configure and run your application without any XML (aside from the MyBatis query definitions). This post is a long overdue follow up to a previous post about my contribution to this code.\n\u0026#43; Please note: The examples shown here work with Spring 4.0.6 and 4.2.4. Check the master branch on GitHub for updates to the version of Spring compatible with these examples.\nA Java Config Appetizer #There is a lot of conflicting information out there for those searching for how to implement Spring\u0026rsquo;s Java Config. Be sure to check the version of Spring used in the example because it may not match your target version. This example uses Spring Framework 4.0.6 and MyBatis 3.2.7. As not to get into the weeds of Java Config for a Spring MCV application, we\u0026rsquo;ll cover just the pertinent parts to integrating MyBatis with Java Config.\nHave a look at the file structure below. The AppInitializer with its AbstractAnnotationConfigDispatcherServletInitializer super class is where life begins for the application. The getRootConfigClasses() method returns the DataConfg class amongst others not pictured below.\nFile structure:\n- src/main - java/org/lanyonm/playground - config * AppInitializer.java * DataConfig.java - domain * User.java - persistence * UserMapper.java - resources/org/lanyonm/playground - persistence * UserMapper.xml * pom.xml It\u0026rsquo;s my preference to put all the @Component or @Configuration classes into the config package so it\u0026rsquo;s easy to locate where the application components are configured.\nThe Key Files #There are four main files in this example. The first and most important is DataConfig.java because it\u0026rsquo;s where the @MapperScan annotation is used. On line 2, you see the package where the MyBatis mappers reside. The three @Bean annotated methods provide the Java Config equivalent to what you would typically see in xml configuration for MyBatis. In this case a SimpleDriverDataSource is used in place of a full-blown DataSource and specifies an in-memory H2 database. In future iterations of this application I will show how to use Spring\u0026rsquo;s Profiles to specify different DataSource implementations depending on the environment.\norg.lanyonm.playground.config.DataConfig.java :\n@Configuration @MapperScan(\u0026#34;org.lanyonm.playground.persistence\u0026#34;) public class DataConfig { @Bean public DataSource dataSource() { SimpleDriverDataSource dataSource = new SimpleDriverDataSource(); dataSource.setDriverClass(org.h2.Driver.class); dataSource.setUsername(\u0026#34;sa\u0026#34;); dataSource.setUrl(\u0026#34;jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE\u0026#34;); dataSource.setPassword(\u0026#34;\u0026#34;); // create a table and populate some data JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource); System.out.println(\u0026#34;Creating tables\u0026#34;); jdbcTemplate.execute(\u0026#34;drop table users if exists\u0026#34;); jdbcTemplate.execute(\u0026#34;create table users(id serial, firstName varchar(255), lastName varchar(255), email varchar(255))\u0026#34;); jdbcTemplate.update(\u0026#34;INSERT INTO users(firstName, lastName, email) values (?,?,?)\u0026#34;, \u0026#34;Mike\u0026#34;, \u0026#34;Lanyon\u0026#34;, \u0026#34;lanyonm@gmail.com\u0026#34;); return dataSource; } @Bean public DataSourceTransactionManager transactionManager() { return new DataSourceTransactionManager(dataSource()); } @Bean public SqlSessionFactoryBean sqlSessionFactory() throws Exception { SqlSessionFactoryBean sessionFactory = new SqlSessionFactoryBean(); sessionFactory.setDataSource(dataSource()); sessionFactory.setTypeAliasesPackage(\u0026#34;org.lanyonm.playground.domain\u0026#34;); return sessionFactory; } } The second important piece of DataConfig is on line 32 where we set the package containing the domain objects that will be available as types in the MyBatis xml files. Returning Java objects from SQL queries is why we go to all this ORM trouble after all.\nThe User domain object is really just a simple POJO. I\u0026rsquo;ve omitted the getters and setters for brevity.\norg.lanyonm.playground.domain.User.java :\npublic class User implements Serializable { private static final long serialVersionUID = 1L; private long id; private String firstName; private String lastName; private String email; // getters and setters } MyBatis Mapper classes are simple interfaces with method definitions that match with a sql statement defined in the corresponding mapper xml. It is possible to write simple sql statements in annotations instead of defining the sql in xml, but the syntax becomes cumbersome quickly and doesn\u0026rsquo;t allow for complex queries.\norg.lanyonm.playground.persistence.UserMapper.java :\npublic interface UserMapper { /** * @return all the users */ public List\u0026lt;User\u0026gt; getAllUsers(); /** * @param user * @return the number of rows affected */ public int insertUser(User user); /** * @param user * @return the number of rows affected */ public int updateUser(User user); } I haven\u0026rsquo;t done anything special with the MyBatis xml, just a few simple statements.\norg.lanyonm.playground.persistence.UserMapper.xml :\n\u0026lt;!DOCTYPE mapper PUBLIC \u0026#34;-//mybatis.org//DTD Mapper 3.0//EN\u0026#34; \u0026#34;http://mybatis.org/dtd/mybatis-3-mapper.dtd\u0026#34;\u0026gt; \u0026lt;mapper namespace=\u0026#34;org.lanyonm.playground.persistence.UserMapper\u0026#34;\u0026gt; \u0026lt;cache /\u0026gt; \u0026lt;select id=\u0026#34;getAllUsers\u0026#34; resultType=\u0026#34;User\u0026#34;\u0026gt; SELECT id, firstName, lastName, email FROM users \u0026lt;/select\u0026gt; \u0026lt;insert id=\u0026#34;insertUser\u0026#34; parameterType=\u0026#34;User\u0026#34;\u0026gt; INSERT INTO users (firstName, lastName, email) VALUES (#{firstName}, #{lastName}, #{email}) \u0026lt;/insert\u0026gt; \u0026lt;update id=\u0026#34;updateUser\u0026#34; parameterType=\u0026#34;User\u0026#34;\u0026gt; UPDATE users SET firstName = #{firstName}, lastName = #{lastName}, email = #{email} WHERE ID = #{id} \u0026lt;/update\u0026gt; \u0026lt;/mapper\u0026gt; That\u0026rsquo;s pretty much all there is to it. If you find something I left out, please let me know. The full source code for this example resides in the playground repo on GitHub.\n","date":"2014-04-21","permalink":"/articles/spring-4-mybatis-java-config/","section":"Articles","summary":"","title":"Spring 4 and MyBatis Java Config"},{"content":"","date":null,"permalink":"/tags/apache/","section":"Tags","summary":"","title":"Apache"},{"content":"","date":null,"permalink":"/tags/grok/","section":"Tags","summary":"","title":"Grok"},{"content":"Update: The version of Logstash used in the example is out of date, but the mechanics of the multiline plugin and grok parsing for multiple timestamps from Tomcat logs is still applicable. I have published a new post about other methods for getting logs into the ELK stack.\nAdditionally, the multiline filter used in these examples is not threadsafe. I have an updated example using the multiline codec with the same parsers in the new post.\nOnce you\u0026rsquo;ve gotten a taste for the power of shipping logs with Logstash and analyzing them with Kibana, you\u0026rsquo;ve got to keep going. My second goal with Logstash was to ship both Apache and Tomcat logs to Elasticsearch and inspect what\u0026rsquo;s happening across the entire system at a given point in time using Kibana. Most of the apps I write compile to Java bytecode and use something like log4j for logging. The logging isn\u0026rsquo;t always the cleanest and there can be several conversion patterns in one log.\nKibana showing Apache and Tomcat responses for a 24 hour period (at a 5 minute granularity). Log Format #Parsing your particular log\u0026rsquo;s format is going to be the crux of the challenge, but hopefully I\u0026rsquo;ll cover the thought process in enough detail that parsing your logs will be easy.\nApache Logs #The Apache log format is the default Apache combined pattern (\u0026quot;%h %l %u %t \\\u0026quot;%r\\\u0026quot; %\u0026gt;s %b \\\u0026quot;%{Referer}i\\\u0026quot; \\\u0026quot;%{User-Agent}i\\\u0026quot;\u0026quot;):\n12.34.56.78 - - [09/Jan/2014:04:02:26 -0800] \u0026#34;GET / HTTP/1.1\u0026#34; 200 43977 \u0026#34;-\u0026#34; \u0026#34;Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)\u0026#34; 123.4.56.7 - - [09/Jan/2014:04:02:26 -0800] \u0026#34;GET /financing/incentives HTTP/1.1\u0026#34; 200 20540 \u0026#34;https://www.google.com/\u0026#34; \u0026#34;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.41 Safari/537.36\u0026#34; 123.45.67.89 - - [09/Jan/2014:04:02:28 -0800] \u0026#34;GET /static/VEAMUNaPbswx4l9JeZqItoN6YKiVmYY84EJKnPKSPPM.css HTTP/1.1\u0026#34; 200 6497 \u0026#34;http://www.example.com/financing/incentives\u0026#34; \u0026#34;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.41 Safari/537.36\u0026#34; 123.45.67.89 - - [09/Jan/2014:04:02:28 -0800] \u0026#34;GET /static/u5Uj9e2Cc98JPkk2CIHl4SGWcqeQ0YU9O7Ua61z9Qdi.js HTTP/1.1\u0026#34; 200 6192 \u0026#34;http://www.example.com/financing/incentives\u0026#34; \u0026#34;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.41 Safari/537.36\u0026#34; 123.45.67.89 - - [09/Jan/2014:04:02:28 -0800] \u0026#34;GET /static/ZiQCg9sERShna8pay7mOZZdbUwqH6n6s9bbmpJhOzpo.css HTTP/1.1\u0026#34; 200 626 \u0026#34;http://www.example.com/financing/incentives\u0026#34; \u0026#34;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.41 Safari/537.36\u0026#34; Tomcat Logs #The Tomcat log format in this example is a bit more mixed, with a combination of Tomcat\u0026rsquo;s SimpleFormatter and a customized Log4j conversion pattern (\u0026quot;%d{yyyy-MM-dd HH:mm:ss,SSS ZZZ} | %p | %c - %m%n\u0026quot;). Here\u0026rsquo;s an example of the combined log:\nJan 9, 2014 7:13:13 AM org.apache.tomcat.util.http.Parameters processParameters INFO: Character decoding failed. Parameter [dcp] with value [ppn.%epid!.] has been ignored. Note that the name and value quoted here may be corrupted due to the failed decoding. Use debug level logging to see the original, non-corrupted values. Note: further occurrences of Parameter errors will be logged at DEBUG level. 2014-01-09 17:32:25,527 -0800 | ERROR | com.example.controller.ApiController - Request exception javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://api.example.com/DataServices/Data?WSDL. It failed with: Connection reset. at com.example.webservices.Data.\u0026lt;init\u0026gt;(Data.java:50) at com.example.service.soap.DataService.submitRequest(DataService.groovy:28) at com.example.service.request.RequestService.addRequest(RequestService.groovy:26) at com.example.controller.ApiController.request(ApiController.groovy:692) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:200) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:311) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898) Caused by: java.net.SocketException: Connection reset ... 17 more This is a somewhat arbitrary non-default conversion pattern, but I\u0026rsquo;ll go into greater detail below on the parsing details as well as providing some handy resources on building pattern matchers.\nThe Logstash Config #To understand the filter section, we must first have a look at the input. I defined four tcp inputs because I piped logs from four different servers into Logstash and wanted to be able to label them as such. As you can see below, each input adds a \u0026quot;server\u0026quot; field that identifies which server the log came from (given other circumstances, this may not be necessary):\ninput { tcp { type =\u0026gt; \u0026#34;apache\u0026#34; port =\u0026gt; 3333 add_field =\u0026gt; { \u0026#34;server\u0026#34; =\u0026gt; \u0026#34;prod1\u0026#34; } } tcp { type =\u0026gt; \u0026#34;apache\u0026#34; port =\u0026gt; 3334 add_field =\u0026gt; { \u0026#34;server\u0026#34; =\u0026gt; \u0026#34;prod2\u0026#34; } } tcp { type =\u0026gt; \u0026#34;tomcat\u0026#34; port =\u0026gt; 3335 add_field =\u0026gt; { \u0026#34;server\u0026#34; =\u0026gt; \u0026#34;prod1\u0026#34; } } tcp { type =\u0026gt; \u0026#34;tomcat\u0026#34; port =\u0026gt; 3336 add_field =\u0026gt; { \u0026#34;server\u0026#34; =\u0026gt; \u0026#34;prod2\u0026#34; } } } Use the following Netcat command with TCP inputs and local log files: nc localhost 3333 \u0026lt; prod1/access.log. I realize that the pipe input would have worked as well, and if we were running this on a production system the configuration would be different, but I\u0026rsquo;ll address that later.\nFilter Config #There\u0026rsquo;s quite a bit of nuance in the filter config that was not immediately apparent to me. First off, in the most recent versions of Logstash, the if/elseif/else logic is preferred to the grep filter. There are a lot of great examples on the web that haven\u0026rsquo;t been updated to use the new convention.\nThe Apache processing is something I\u0026rsquo;ve detailed in a previous post , but it is important to note the added date filter. This filter helps Logstash understand the exact time the event occurred. You\u0026rsquo;ll notice that the time format matches the timestamp in the Apache logs.\nfilter { if [type] == \u0026#34;apache\u0026#34; { grok { patterns_dir =\u0026gt; \u0026#34;/Users/lanyonm/logstash/patterns\u0026#34; match =\u0026gt; { \u0026#34;message\u0026#34; =\u0026gt; \u0026#34;%{COMBINEDAPACHELOG}\u0026#34; } } date { match =\u0026gt; [ \u0026#34;timestamp\u0026#34;, \u0026#34;dd/MMM/yyyy:HH:mm:ss Z\u0026#34; ] } } ... The multiline filter is the key for Logstash to understand log events that span multiple lines. In my case, each Tomcat log entry began with a timestamp, making the timestamp the best way to detect the beginning of an event. The challenge was that there were multiple timestamp formats. Here are two examples:\nJan 9, 2014 7:13:13 AM 2014-01-09 17:32:25,527 -0800 These weren\u0026rsquo;t entirely standard patterns, so I had to customize grok patterns to match. The following patterns can be found in my grok-patterns gist:\nCATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM) TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE} Using these two patterns, we are able to construct the multiline pattern to match both conversion patterns. The negate and previous mean that each line will log-line rolls into the previous lines unless the pattern is matched.\nif [type] == \u0026#34;tomcat\u0026#34; { multiline { patterns_dir =\u0026gt; \u0026#34;/Users/lanyonm/logstash/patterns\u0026#34; pattern =\u0026gt; \u0026#34;(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})\u0026#34; negate =\u0026gt; true what =\u0026gt; \u0026#34;previous\u0026#34; } } The next thing to do is parse each event into its constituent parts. In the Tomcat log example above, the timestamp is followed by a logging level, classname and log message. Grok already provides some of of these patterns, so we just had to glue them together. Again, because there are two different syntaxes for a log statement, we have two patterns:\nCATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage} TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \\| %{LOGLEVEL:level} \\| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage} We see some of the filter nuance below. These two patterns can be checked against an event by specifying the match with a hash of comma-separated keys and values. The grok filter will attempt to match each pattern before failing to parse. The filter\u0026rsquo;s match documentation isn\u0026rsquo;t quite perfected on this point yet. Have a look at the grok filter below:\nif \u0026#34;_grokparsefailure\u0026#34; in [tags] { drop { } } grok { patterns_dir =\u0026gt; \u0026#34;/Users/lanyonm/logstash/patterns\u0026#34; match =\u0026gt; [ \u0026#34;message\u0026#34;, \u0026#34;%{TOMCATLOG}\u0026#34;, \u0026#34;message\u0026#34;, \u0026#34;%{CATALINALOG}\u0026#34; ] } date { match =\u0026gt; [ \u0026#34;timestamp\u0026#34;, \u0026#34;yyyy-MM-dd HH:mm:ss,SSS Z\u0026#34;, \u0026#34;MMM dd, yyyy HH:mm:ss a\u0026#34; ] } Inevitably, there will be mess in your logs that doesn\u0026rsquo;t conform to your grok parser. You can choose to drop events that fail to parse by using the drop filter inside a conditional as shown on the second line above. Where do [tags] come from you might ask? Tags can be applied to events at several points in the processing pipeline. For example, when the multiline filter successfully parses an event, it tags the event with \u0026quot;multiline\u0026quot;.\nYou can also see that the date filter can accept a comma separated list of timestamp patterns to match. This allows either the CATALINA_DATESTAMP pattern or the TOMCAT_DATESTAMP pattern to match the date filter and be ingested by Logstash correctly.\nOutput #The output is simply an embedded Elasticsearch config as well as debugging to stdout. If you\u0026rsquo;d like to see the full config, have a look at the gist .\nGrok Patterns #There\u0026rsquo;s no magic to grok patterns (unless the built-ins work for you). There are however a couple resources that can make your parsing go faster. First is the Grok Debugger . You can paste messages into the Discover tab and the Debugger will find the best matches against the built in patterns. Another regex assistant I use is RegExr . I have the native app, but the web page is nice too.\nThe full list of patterns shipped with Logstash can be found on GitHub , and the ones I used can be found in this Gist . If you\u0026rsquo;re not into clicking links, here are the important ones:\nJAVACLASS (?:[a-zA-Z0-9-]+\\.)+[A-Za-z0-9$]+ JAVALOGMESSAGE (.*) # MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM) # yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800 TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE} CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage} # 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened... TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \\| %{LOGLEVEL:level} \\| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage} The Kibana Dashboard #As I mentioned at the top, the goal of this endeavor was to be able to correlate Apache and Tomcat logs. We often find ourselves asking what a user had been doing on the website when he or she encountered a server error. Elasticsearch and Kibana can put all logs on the same timeline. The production system whose logs I used for experimentation has a pair of servers, each hosting a Tomcat instance and an Apache instance whose logs are divided between static (CDN cached) and non-static requests. I configured a Kibana dashboard so it would display the static and non-static web requests separately as well as separate the application logs per server.\nThe Kibana Dashboard (click/tap to enlarge) You can see the obvious red and orange areas where a deploy rolled through the system. The colors could be further tweaked to show API calls vs. html pages or to further break down the static content into its constituent mime types. The json representation of the dashboard is here .\nIf you\u0026rsquo;re new to Kibana and you\u0026rsquo;d like to use this dashboard, you can download the json and from the Kibana UI and load the dashboard from disk using the json.\nTry It Yourself #I wrote a handy script that can be used in conjunction with other files in the gist :\nIt should be as easy as ./logstash.sh. If you\u0026rsquo;re testing out new patterns for your particular log format I would suggest commenting out the embedded Elasticsearch output and the -- web (which runs Kibana) from the shell script.\nIn Production #One caveat I\u0026rsquo;d like to make is that the configurations I\u0026rsquo;ve presented here would not be suitable in production. For example, you would want to use a standalone Elasticsearch instance. The config would also be simpler because each log shipper would be on its respective server and the input would likely be a file . You could easily make an argument for a Logstash process per server that information if being collected from as well.\n","date":"2014-01-12","permalink":"/articles/logstash-multiline-tomcat-log-parsing/","section":"Articles","summary":"","title":"Logstash Multiline Tomcat and Apache Log Parsing"},{"content":"","date":null,"permalink":"/tags/tomcat/","section":"Tags","summary":"","title":"Tomcat"},{"content":"","date":null,"permalink":"/tags/graphite/","section":"Tags","summary":"","title":"Graphite"},{"content":"As web app developers, our first experiences pulling data from web server logs is often with piped unix commands - but there\u0026rsquo;s a better way! I leveled up to log data aggregation with Logster , but ultimately moved on to Logstash . Logstash provides a clean and easy to understand DSL for shipping, parsing and stashing logs. What attracted me to Logstash was it\u0026rsquo;s ease of integration into Elasticsearch and the resulting capability to query logs without predefining the queries.\nSending Web Server Response Codes to Graphite #Let\u0026rsquo;s say you want to monitor the counts and distribution of response codes logged by a web server. For the purposes of this example, we\u0026rsquo;ll be sending Nginx response codes to Graphite via StatsD using Logstash. I\u0026rsquo;ll focus on the Logstash configuration, but if you want to stand up Graphite \u0026amp; StatsD quickly, check this out .\nThe Logstash configuration DSL has three main sections: input, filter \u0026amp; output. Think of these as three operations along an event processing pipeline. What each section does is defined by plugins, which can be summarized as data streams or transforms. I use file, grok, statsd and elasticsearch for this example. You can find much more documentation here .\nHere\u0026rsquo;s the logstash.conf file:\nThe input section is pretty self explanatory. The one thing that is important to understand is that type is used in the filter section to determine which filters will take action on that data.\nThe filter section is deceptively simple. Grok is community jargon for comprehensive understanding, but it is also a tool for pattern matching . The COMBINEDAPACHELOG is a grok pattern provided by Logstash and in this case is used to define all the data in the message. I\u0026rsquo;ll go into greater detail below.\nThe output section specifies two destinations with a third commented out. The statsd configuration specifies where the StatsD server lives and what action to take. In this case, it will increment a counter per each host and response code. The elasticsearch output is telling Logstash to send all the data to an embedded Elasticsearch instance. More about that below as well.\nAssuming you have the Logstash jar and the configuration file in the same directory, you can use the following command to run Logstash:\njava -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web The agent parameter tells Logstash to run the inputs, filters and outputs while web runs Kibana. More about Kibana later. You\u0026rsquo;ll notice that it takes a few seconds to start up and that there\u0026rsquo;s some status information on the plugins that we\u0026rsquo;re using. Interestingly, the logging doesn\u0026rsquo;t look anything like logs configured with log4j. That\u0026rsquo;s because Logstash is actually written in Ruby and packaged as a jar with JRuby for portability and ease of install. If you get a really funky stacktrace from the jar that contains Ruby references, that\u0026rsquo;s why.\nIf everything is setup correctly, you\u0026rsquo;ll get info like this in Graphite:\nNginx Response Codes in Graphite for an 84 hour period Pretty nifty, eh? As you can see, this snapshot includes a long period where the primary server response was a 404, which made this time range more interesting than others.\nGrok Parsing #The grok filter is a key part to why this works with so little configuration. I glossed over why COMBINEDAPACHELOG worked for the Nginx access log, so lets have a deeper look. The definition looks like this:\nCOMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent} Ok, so it\u0026rsquo;s COMMONAPACHELOG plus referrer and user agent. COMMONAPACHELOG looks like this:\nCOMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \u0026#34;(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\u0026#34; %{NUMBER:response} (?:%{NUMBER:bytes}|-) What you\u0026rsquo;re looking at is aliased grok patterns and corresponding field names. Each data type is actually just a regex, which you can see in the grok patterns file . For example, IPORHOST resolves to (?:%{HOSTNAME}|%{IP}). HOSTNAME is a rather long regex while IP is further broken down into IPV4 and IPV6.\nIPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)? IPV4 (?\u0026lt;![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9]) IP (?:%{IPV6}|%{IPV4}) HOSTNAME \\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\\b) IPORHOST (?:%{HOSTNAME}|%{IP}) If your needs are more unique, the grok filter documentation shows how to create custom fields or even whole customized pattern files.\nNow let\u0026rsquo;s have a look at an example log entry and see what the Grok filter does with it. Here\u0026rsquo;s the log entry:\n10.0.0.1 - - [28/Nov/2013:11:01:31 -0600] \u0026#34;GET / HTTP/1.1\u0026#34; 200 303 \u0026#34;-\u0026#34; \u0026#34;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/30.0.1599.114 Chrome/30.0.1599.114 Safari/537.36\u0026#34; Notice how the COMBINEDAPACHELOG pattern matches with the syntax of the log entry. If we uncomment the stdout { codec =\u0026gt; rubydebug } output, we can see how the parsed log entry is turned into an event:\n{ \u0026#34;message\u0026#34; =\u0026gt; \u0026#34;10.0.0.1 - - [28/Nov/2013:11:01:31 -0600] \\\u0026#34;GET / HTTP/1.1\\\u0026#34; 200 303 \\\u0026#34;-\\\u0026#34; \\\u0026#34;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/30.0.1599.114 Chrome/30.0.1599.114 Safari/537.36\\\u0026#34;\u0026#34;, \u0026#34;@timestamp\u0026#34; =\u0026gt; \u0026#34;2013-11-28T17:01:32.003Z\u0026#34;, \u0026#34;@version\u0026#34; =\u0026gt; \u0026#34;1\u0026#34;, \u0026#34;type\u0026#34; =\u0026gt; \u0026#34;nginx-access\u0026#34;, \u0026#34;host\u0026#34; =\u0026gt; \u0026#34;lanyonm-linux\u0026#34;, \u0026#34;path\u0026#34; =\u0026gt; \u0026#34;/var/log/nginx/access.log\u0026#34;, \u0026#34;clientip\u0026#34; =\u0026gt; \u0026#34;10.0.0.1\u0026#34;, \u0026#34;ident\u0026#34; =\u0026gt; \u0026#34;-\u0026#34;, \u0026#34;auth\u0026#34; =\u0026gt; \u0026#34;-\u0026#34;, \u0026#34;timestamp\u0026#34; =\u0026gt; \u0026#34;28/Nov/2013:11:01:31 -0600\u0026#34;, \u0026#34;verb\u0026#34; =\u0026gt; \u0026#34;GET\u0026#34;, \u0026#34;request\u0026#34; =\u0026gt; \u0026#34;/\u0026#34;, \u0026#34;httpversion\u0026#34; =\u0026gt; \u0026#34;1.1\u0026#34;, \u0026#34;response\u0026#34; =\u0026gt; \u0026#34;200\u0026#34;, \u0026#34;bytes\u0026#34; =\u0026gt; \u0026#34;303\u0026#34;, \u0026#34;referrer\u0026#34; =\u0026gt; \u0026#34;\\\u0026#34;-\\\u0026#34;\u0026#34;, \u0026#34;agent\u0026#34; =\u0026gt; \u0026#34;\\\u0026#34;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/30.0.1599.114 Chrome/30.0.1599.114 Safari/537.36\\\u0026#34;\u0026#34; } The individual fields can be used in the output section to send select information to tools like StatsD. It\u0026rsquo;s now very clear how increment =\u0026gt; \u0026quot;%{host}.nginx.response.%{response}\u0026quot; gets populated.\nElasticsearch \u0026amp; Kibana #This is where the magic happens. Without going into too much detail, Elasticsearch accepts this semi-structured data and builds searchable indexes for it. It becomes very easy to answer questions like, \u0026ldquo;how many non-2xx level error codes have we gotten per unique IPs?\u0026rdquo; If you want to hear more about how the query language works or how it handles large volumes of data, check out the website . The embedded Elasticsearch instance will persist data in a data folder in the same location Logstash is run.\nKibana is the super-slick web interface built on top of Elasticsearch. It provides that very necessary visualization on top of the json responses that Elasticsearch provides. The web option of the Logstash startup command runs Kibana on port 9292. This is what it looks like:\nNginx log data from Elasticsearch, displayed by Kibana You\u0026rsquo;ll notice how the data is similar to the Graphite picture above. Unlike Graphite, Elasticsearch has indexed all the log data and you can therefore ask questions you had not prepared to answer prior to collecting the data (as would be the constraint with Graphite).\nA tool like Elasticsearch coupled with Kibana makes log data much more operationally potent. For example, you could easily keep an eye on auditd logs for atypical access attempts. Similarly you could watch your website administration pages for IPs outside of the typical users. When used with log4j configured logs, the multiline input plugin will collapse a multi-line stacktrace into a single event. Elasticsearch would then be able to count the occurrences of each error, display frequency, and potentially even geo-locate if you get fancy with your web server logs.\nThe Business Proposition #Logstash and it\u0026rsquo;s companion tools like Elasticsearch and Kibana can transform logs from a stale and sometimes inaccessible disk-space liability into an operational advantage. The log data in the Kibana dashboard can be near real-time and the flexibility of Elasticsearch\u0026rsquo;s query interface makes asking big questions easy. Having this information at your fingertips will give you an operation edge.\n","date":"2013-11-27","permalink":"/articles/pushing-web-server-response-codes-graphite-logstash/","section":"Articles","summary":"","title":"Pushing Web Server Response Codes into Graphite with Logstash"},{"content":"When you host a Jekyll site on GitHub , it renders the site in safe-mode. This means you can\u0026rsquo;t use any plugins enhance Jekyll\u0026rsquo;s functionality and, more specifically, you can\u0026rsquo;t write a Liquid tag that alphabetizes the site\u0026rsquo;s tags. Unordered tags make a Jekyll tag list look sloppy, so if you care about the details, you likely want your tag list to be alphabetized.\nIt\u0026rsquo;s easy to alphabetize the tag list on a given page (just list the tags alphabetically), but getting your tags.html to list tags alphabetically is a bit more convoluted if you aren\u0026rsquo;t able to use plugins.\nWaist-Deep in Liquid #Liquid provides some pretty good documentation for figuring out what logical operators and functions are built into the templating language. Particularly useful are capture and assign, which each capture values as variables. These logical operators can contain additional logic like for-loops, splits, sorts and joins. Here\u0026rsquo;s the code I use to generate a sorted list of tags:\n{% capture site_tags %}{% for tag in site.tags %}{{ tag | first }}{% unless forloop.last %},{% endunless %}{% endfor %}{% endcapture %} {% assign tag_words = site_tags | split:\u0026#39;,\u0026#39; | sort %} Line 1 above will get the tag name for every tag on the site and set them to the site_tags variable. Each tag object contains both the tag name and a list of the associated posts. This capture statement is all on one line so that the commas can reliably be used as delimiters. Line 2 creates the tag_words variable that is a sorted array of the tag names. If you want to see what these two statements would produce within Jekyll, you\u0026rsquo;ll find them printed in html comments on my tags page .\nBuilding The HTML #As you can see on tags.html , the tags are listed alphabetically with post counts and then each tag\u0026rsquo;s post is listed below. Here\u0026rsquo;s the Liquid code:\n\u0026lt;div id=\u0026#34;tags\u0026#34;\u0026gt; \u0026lt;h1\u0026gt;Tags\u0026lt;/h1\u0026gt; \u0026lt;ul class=\u0026#34;tag-box inline\u0026#34;\u0026gt; {% for item in (0..site.tags.size) %}{% unless forloop.last %} {% capture this_word %}{{ tag_words[item] | strip_newlines }}{% endcapture %} \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;#{{ this_word | cgi_escape }}\u0026#34;\u0026gt;{{ this_word }} \u0026lt;span\u0026gt;{{ site.tags[this_word].size }}\u0026lt;/span\u0026gt;\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; {% endunless %}{% endfor %} \u0026lt;/ul\u0026gt; {% for item in (0..site.tags.size) %}{% unless forloop.last %} {% capture this_word %}{{ tag_words[item] | strip_newlines }}{% endcapture %} \u0026lt;h2 id=\u0026#34;{{ this_word | cgi_escape }}\u0026#34;\u0026gt;{{ this_word }}\u0026lt;/h2\u0026gt; \u0026lt;ul class=\u0026#34;posts\u0026#34;\u0026gt; {% for post in site.tags[this_word] %}{% if post.title != null %} \u0026lt;li itemscope\u0026gt;\u0026lt;span class=\u0026#34;entry-date\u0026#34;\u0026gt;\u0026lt;time datetime=\u0026#34;{{ post.date | date_to_xmlschema }}\u0026#34; itemprop=\u0026#34;datePublished\u0026#34;\u0026gt;{{ post.date | date: \u0026#34;%B %d, %Y\u0026#34; }}\u0026lt;/time\u0026gt;\u0026lt;/span\u0026gt; \u0026amp;raquo; \u0026lt;a href=\u0026#34;{{ post.url }}\u0026#34;\u0026gt;{{ post.title }}\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; {% endif %}{% endfor %} \u0026lt;/ul\u0026gt; {% endunless %}{% endfor %} \u0026lt;/div\u0026gt; On line 4 you\u0026rsquo;ll notice an unless forloop.last. This is because it\u0026rsquo;s a quick an easy way to keep the array index in bounds. Lines 6 and 14 show how you can use a tag\u0026rsquo;s name to pull the tag from site.tags. I checked for post.title != null because I don\u0026rsquo;t want to display pages without titles.\nTL;DR #Alphabetizing Jekyll site tags is trickier than it should be. If you plan to deploy the site via GitHub Pages, you can\u0026rsquo;t use plugins. The source code for my pure-Liquid, alphabetized tags is on GitHub and deployed here .\n","date":"2013-11-21","permalink":"/articles/alphabetize-jekyll-page-tags-pure-liquid/","section":"Articles","summary":"","title":"Alphabetizing Jekyll Page Tags In Pure Liquid (Without Plugins)"},{"content":"","date":null,"permalink":"/tags/hiking/","section":"Tags","summary":"","title":"Hiking"},{"content":"","date":null,"permalink":"/tags/national-parks/","section":"Tags","summary":"","title":"National Parks"},{"content":"","date":null,"permalink":"/tags/vacation/","section":"Tags","summary":"","title":"Vacation"},{"content":"","date":null,"permalink":"/tags/yosemite/","section":"Tags","summary":"","title":"Yosemite"},{"content":"I\u0026rsquo;ve always loved the outdoors. My parents took me to a handful of the National Parks , but with 59 parks in total we couldn\u0026rsquo;t see everything. Yosemite , only a few hours from San Francisco, is easily accessible by car and ripe for a long weekend getaway. Erin and I visited Oct, 23 - 27 for 3 full days of wilderness fun.\nLooking west into Yosemite Valley from the top of North Dome Late October is off-season for Yosemite, but we didn\u0026rsquo;t really know how much that would limit us - turns out, not at all. With nothing but good weather in the forecast, all options were on the table. The owners of the vacation rental we stayed at, Tom and Theresa, are avid climbers and gave us the scoop on how to make the best of our three days.\nMaximizing Our Time #Theresa gave us several options, but given that we only had three full days, she prioritized these three as must-see:\nMist Trail - Vernal Falls \u0026amp; Nevada Falls Mariposa Grove \u0026amp; Glacier Point North Dome \u0026amp; Indian Rock The hike up to Nevada Falls and the hike from Tioga Road to North Dome are both on the more strenuous side, so we did those first and last. Mariposa Grove and Glacier Point are separated by about an hour car ride, so that made for an easier middle day. We knew we would have a few hours the morning before we drove back to San Fran, so we used that time to talk a leisurely walk in the valley meadows.\nThe Hikes #Instead of going deep into our experience of each hike (and all the pictures), I\u0026rsquo;ll instead provide a quick synopsis and map. Hopefully I can find the time to put more pictures online.\nVernal Falls \u0026amp; Nevada Falls # The Hike from Happy Isles to Nevada Falls - the valley is up and to the left of this map It would have been wise to start out easier than we did, so the altitude and incline reminded us. The hike includes several very steep sections that are in no way technically difficult, but are certainly physically strenuous for those unaccustomed to the altitude. Take short breaks to allow yourself to recover and to take in the view. There\u0026rsquo;s several great spots at Vernal Falls to refuel, but if you plan to head up to Nevada Falls, I suggest saving lunch till then.\nA View of Liberty Cap from Emerald Pool just above Vernal Falls Even in October, with the lowest water levels of the season, the Merced River is impressive as it pounds down the falls. On our way down we came across a Bobcat near the top of Vernal Falls. It was pretty exciting. I can\u0026rsquo;t be certain, but I think I prefer less water in the falls instead of 3 or 4 times as many people on the trail.\nMariposa Grove \u0026amp; Glacier Point # Mariposa Grove Map Hiking in the grove is an pleasant, shaded relaxation compared to the stone steps along Mist Trail. The Giant Sequoias in the grove are astounding. No picture I\u0026rsquo;ve seen (and certainly not ones I\u0026rsquo;ve taken) can do their enormity justice. It\u0026rsquo;s truly something that must be seen in person.\nErin inside the California Tunnel Tree in Mariposa Grove If you have some spring in your step, I suggest taking the Outer Loop Trail as well as the detour out to Wawona Point Vista. The Vista is a perfect spot for lunch. The fallen Wawona Tunnel Tree and Telescope Tree are further into the Grove, and are must-sees. Also, the return trip to the parking lot will be mostly downhill, so there\u0026rsquo;s no need to save energy.\nThe best way to end a day at Mariposa Grove is to spend an hour at Glacier Point. Be sure to arrive about at least 30 minutes before sunset so you can experience the color change from gold to pink.\nThe sun beginning to set on North Dome and Half Dome - taken from Glacier Point Glacier Point\u0026rsquo;s elevation is ~7200 ft, so it was cold as the sun set. Bundle up so you can appreciate the view as long as you like.\nNorth Dome # North Dome Trail Map The hike from Tioga Road to North Dome and Indian Rock was stunning. As you hike toward the valley rim you are exposed to ever expansive vistas of the surrounding domes, caps, and eventually the valley floor. The photo at the top of this post is looking westward from North Dome with just a bit of the road on the valley floor visible (zoom in to see the vehicles parked along the road). North Dome also provides the best views of Half Dome.\nA view of Half Dome from North Dome Indian Rock is a short and worthwhile detour from the main trail. The arch is much smaller than what you\u0026rsquo;d see in Arches National Park, but when you think about how all of Yosemite was formed by Glaciers and that this granite arch is hundreds of feet above its surroundings, it\u0026rsquo;s pretty amazing.\nIndian Rock Archway Don\u0026rsquo;t forget that the last 3/4 of a mile back to Tioga Road are uphill. ;)\nCook\u0026rsquo;s Meadow #We had an hour or two before we had to head back to San Francisco and we couldn\u0026rsquo;t have chosen a better place to spend the time. There was stunning beauty every direction.\nA view of Yosemite Falls from Cook's Meadow With the advice of an experienced hiker, Yosemite wasn\u0026rsquo;t nearly as daunting as it would have otherwise been. Erin and I feel that we made the most of our time and couldn\u0026rsquo;t be happier with the trip. I hope to upload more photos to Flickr in the coming weeks.\n","date":"2013-11-10","permalink":"/articles/yosemite-three-hike-days/","section":"Articles","summary":"","title":"Yosemite for 3 Days of Hiking"},{"content":"","date":null,"permalink":"/tags/grails/","section":"Tags","summary":"","title":"Grails"},{"content":"","date":null,"permalink":"/tags/testing/","section":"Tags","summary":"","title":"Testing"},{"content":"Maintaining plugins for quickly evolving frameworks can become burdensome if the testing tools don\u0026rsquo;t help you ensure backward compatibility. We have been slowly incorporating better testing into the Grails Feature Toggle Plugin and have been looking to test all minor versions of Grails 2. If only there were a way for Travis-CI to test all those versions of Grails…\n\u0026#43; Thanks to bjfish , there\u0026rsquo;s a quick and easy way to have Travis test multiple versions of Grails. Using GVM, you can install an environment-specified version of Grails within Travis. We have successfully integrated this approach into the Feature Toggle Plugin. Have a look at our .travis.yml here:\nlanguage: groovy jdk: - oraclejdk6 env: - GRAILS_VERSION=2.3.1 - GRAILS_VERSION=2.2.4 - GRAILS_VERSION=2.1.5 - GRAILS_VERSION=2.0.4 before_install: - rm -rf ~/.gvm - curl -s get.gvmtool.net \u0026gt; ~/install_gvm.sh - chmod 775 ~/install_gvm.sh - ~/install_gvm.sh - echo \u0026#34;gvm_auto_answer=true\u0026#34; \u0026gt; ~/.gvm/etc/config - source ~/.gvm/bin/gvm-init.sh - gvm install grails $GRAILS_VERSION || true branches: only: - master - more-testing script: grails clean \u0026amp;\u0026amp; grails upgrade --non-interactive \u0026amp;\u0026amp; grails test-app --non-interactive Another likely influential post outlines this same approach and the comments (this one in particular ) discuss whether this is the cleanest solution. It certainly sounds like something to look into.\nHopefully using Travis-CI to test multiple versions of Grails will help maintainers establish backward compatibility for their plugins and libraries.\n","date":"2013-10-21","permalink":"/articles/testing-multiple-grails-versions-travis-ci/","section":"Articles","summary":"","title":"Testing With Multiple Grails Versions on Travis-CI"},{"content":"","date":null,"permalink":"/tags/hardware-hacking/","section":"Tags","summary":"","title":"Hardware Hacking"},{"content":"","date":null,"permalink":"/tags/xbee/","section":"Tags","summary":"","title":"Xbee"},{"content":"For a few months I have had the parts to build another transmitter for my XBee / Kill-a-Watt power measuring system . Since I\u0026rsquo;ve already chronicled the overall system but didn\u0026rsquo;t have any photos of the process, this post will provide the pretty pictures I lacked before.\nThe Parts #I started with parts from Digi-Key and Adafruit. At the outset, you don\u0026rsquo;t start with much.\nThe Adafruit XBee Adapter Kit and Kill-a-Watt. All the parts aren\u0026rsquo;t pictured above, but here\u0026rsquo;s the full parts list (save a few resistors, heat shrink and ribbon cable):\nAdafruit XBee Adapter Kit XBee Series 1 with U.FL connector 2.4 GHz RF Antenna P3 Kill-a-Watt 50V, 1A Diode 220UF 6.3V Capacitor 10000UF, 6.3V Capacitor Green LED It would have been easier to purchase the add-on kit from Adafruit, but I had enough parts lying around and wanted to get the XBee module with the U.FL connector to as receiver XBee. After the initial build I found that the XBee Series 1 that ship with the Adafruit kits could barely communicate between adjacent rooms.\nThe Easy Part #There are several small parts and headers you solder onto the XBee adapter board before you start doing more hacky stuff. Here\u0026rsquo;s a picture at about that point.\nXBee adapter board with the easy soldering complete. You can see how short those 10-pin headers are from the first picture. Getting these parts to fit under that height is easy, but the next parts, no so much.\nFinishing the Transmitter #The next several steps are kinda messy. You need to solder second resistors onto two already on the board and the 220uf capacitor gets bent underneath where the XBee module will sit. The leads coming from those resistors get soldered onto the leads from the 10000UF capacitor. The diode gets added to the positive terminal of the capacitor and everything gets covered in heat shrink.\nXBee parts soldered and ready to go into the Kill-a-Watt. Instead of separated ribbon cable I used solid wire. If I make another transmitter, I would use braided wire (or separated ribbon cable) due to it\u0026rsquo;s flexibility.\nIntegrating into the Kill-a-Watt #The next step is to open up the Kill-a-Watt and figure out whether you have a P4400 or a P4400.1. The later has a smaller version of the 2902 op-amp which is located behind the Kill-a-Watt display. The leads for the display can be seen in a line between the two clips in the photo below. the display must be flush against these for the Kill-a-Watt display to function correctly, and with the tight fit, getting the leads from the op-amp around the board to the backside of the Kill-a-Watt housing can be a challenge. It\u0026rsquo;s difficult to see in the picture below, but I shaved a millimeter or so off the left side of the PCB. The \u0026ldquo;R\u0026rdquo; in R8 is partly shaven off.\nThe lead wires from the XBee soldered to the 2902 Op-Amp. I also used a razor blade and chisel to whittle away some of the inside of the plastic Kill-a-Watt housing. It\u0026rsquo;s difficult to see, but absolutely essential to being able to re-seat the PCB in a way that allows the display leads to connect correctly. There are two supporting plastic pieces in the middle of the right side of the PCB, which is the best spot to whittle the plastic. Additionally, I recommend using something like 3M Command Hook backing tape to adhere the XBee and capacitor to the housing so they don\u0026rsquo;t rattle around.\nThe XBee transmitter ready to be sealed into the Kill-a-Watt. I would recommend testing before you seal up the unit. Also, don\u0026rsquo;t forget to program the XBee to have a different ID than any other transmitters you may have.\nAll Done! # Fully assembled XBee Kill-A-Watt transmitter. The third-hand tool gives the transmitter one last hug before the transmitter goes into service.\n","date":"2013-10-06","permalink":"/articles/xbee-kill-a-watt-transmitter/","section":"Articles","summary":"","title":"XBee Kill-a-Watt Transmitter"},{"content":"Update #This no longer works with recent versions of the Heavy Water Graphite cookbook. With the release of version 1.0 HW has rewritten the graphite frontend out of their cookbook. If you would still like to follow this example, please be aware that I\u0026rsquo;ve now pinned an old version of the Graphite cookbook.\nTL;DR #I created a Vagrantfile and Cheffile to allow quick and easy creation of a Graphite and StatsD server. You can find the Gist here . Assuming you have the following installed: Vagrant 1.3+, Ruby 1.9+ and the librarian-chef gem, you should be able to run the following commands from the folder containing the Vagrantfile and Cheffile from the gist:\n$ librarian-chef install $ vagrant plugin install vagrant-vbguest $ vagrant plugin install vagrant-omnibus $ vagrant up You should see Vagrant download the box, install Guest Additions, install Omnibus Chef, and provision the Vagrant box. If you encounter problems, let me know so I can add whatever I left out.\nWhy? #I like graphs - no really, I really do. Graphite provides a fantastic way to collect metrics and has a passable dashboard creator. StatsD can be used as a buffering and aggregating proxy for Graphite. Vagrant is the best thing since sliced bread in the local virtual machine management realm. There are plenty of articles that about each of these tools, so follow the links (or Google them) if you\u0026rsquo;re interested in learning more.\nAnother point I want to make clear is that while I use an Ubuntu 12.04 64-bit VirtualBox Vagrant box, because the box is provisioned with a configuration management tool (Chef in this case), these could easily have been changed to CentOS 6.4 or a VMWare image.\nHow - the Chef Parts #I\u0026rsquo;m not going to try to explain Chef in detail, but let\u0026rsquo;s just say that it provides an abstraction layer that allows you to address varied systems in a uniform way. The definition for installing something (like Java or Vim) is encapsulated into a recipe. Multiple recipes for installing the same thing are typically rolled up into a single cookbook. An example of this would be installing Apache from src or from a package. Both those recipes would be in the Apache cookbook.\nTake it a few steps further and you can define how to install Graphite and StatsD on a server and configure them they way you\u0026rsquo;d like. A given cookbook can include recipes from other cookbooks. For example, the StatsD cookbook installs NodeJs.\nLibrarian-Chef provides a way to resolve your cookbook dependencies based on a Cheffile. Here\u0026rsquo;s my Cheffile:\nIt\u0026rsquo;s pretty straightforward. The DSL allows for pretty much anything I\u0026rsquo;ve ever wanted to do.\nHow - the Vagrant Parts #Like I said, Vagrant is the best thing since sliced bread. The Vagrant website does a great job of explaining it, so I won\u0026rsquo;t do any of that. My Vagrantfile does several things:\nSpecifies which Vagrant box to build upon Maps some ports Tells the guest VM to attempt to bridge to the host\u0026rsquo;s network Sets the memory allocation to 1GB Installs Chef 11.6.0 via the Omnibus packaging Installs VirtualBox Guest Additions Installs Vim Installs Graphite and configures it to listen for all traffic to port 2003 Installs StatsD Before I get into any of the details, here\u0026rsquo;s the code:\nThe code above shows how to override some cookbook attributes as well as how to set ports to forward UDP instead of TCP. If you have any questions, please leave a comment.\nPiece Of Mind #Another thing to mention about Vagrant and Chef is that they allow you to create and destroy VMs reliably and reproducibly. If the server were to fail and become irrecoverable, I can use this code that defines infrastructure to build a new one without expending any cognitive effort. Obviously it won\u0026rsquo;t recover or replace your data, but Vagrant never purported to being a backup system. ;)\n","date":"2013-09-15","permalink":"/articles/vagrant-graphite-statsd-chef/","section":"Articles","summary":"","title":"Graphite \u0026 StatsD in Vagrant with Chef"},{"content":"","date":null,"permalink":"/tags/vagrant/","section":"Tags","summary":"","title":"Vagrant"},{"content":"A few weeks ago I went to a Slate Gabfest recording in Chicago during which John Dickerson discussed the longing for a summer camp-like experience for adults. The panel felt that what made camp so special is that it is a truly immersive experience where activities push your boundaries, social interactions are constant, and experiences are formative. Campers have independence from typical authority figures and are freed from normal constraints (for more about camp, listen here ). The Gabfest panel posed the question of whether conferences are a summer camp-like experience for adults - the audience laughed. I shook my head, feeling bad for most everyone in the audience, and as I sit here in one of my DevOpsDays tee-shirts, I\u0026rsquo;ll explain why.\nIf you know me, you know I\u0026rsquo;m into DevOps (a term I\u0026rsquo;ve come to dislike quite a bit for reasons I won\u0026rsquo;t go into here ). I\u0026rsquo;ve been to a few DevOpsDays conferences and to me, they evoke the same feelings of immersion and enthusiasm as camp. This is epitomized by open spaces , where ideas are proposed, discussed and people float freely between sessions. The best way I have found to summarize DevOps is CAMS : culture, automation, measurement and sharing. It is these four things I reflect upon when leading my teams, and all good technology organizations or IT departments share these principles.\nFor the uninitiated, practicing CAMS will change your professional life.\nThe Culture is based on federated responsibility, open communication, and safe experimentation. Automation removes the human hand in routine operations and enables fast feedback loops. Measurement is all about situational awareness, providing the baseline data as well as gauging change. Sharing amplifies the communication aspect of culture - discuss successes, failures, facts and theories. CAMS creates an immersive experience very similar to camp. I know that means that some people will hate it, but, just like camp it will enable the creation of the formative experiences and solutions of its time. Being part of a DevOps organization can be intense. All the metrics of your application are at your fingertips and you can push code through the deploy pipeline to production in under an hour supported by a testing grid running the full suite of functional tests. All technical team members can trigger deploys and are encouraged to share the success (or failure) after features go public. Suffice it to say, it feels pretty free from the normal constraints.\nI agree with the Gabfest crowd that time away from work is necessary to recharge, but being part of a DevOps organization can be just as energizing. Unfortunately, the term DevOps has been usurpted and turned into marketing hype about tooling, but it\u0026rsquo;s equally about culture - a culture that can be as intense, immersive and rewarding experience as those had at camp.\n","date":"2013-08-03","permalink":"/articles/cams-summer-camp/","section":"Articles","summary":"","title":"CAMS Summer Camp"},{"content":"","date":null,"permalink":"/tags/summer-camp/","section":"Tags","summary":"","title":"Summer Camp"},{"content":"","date":null,"permalink":"/tags/mysql/","section":"Tags","summary":"","title":"Mysql"},{"content":"","date":null,"permalink":"/tags/wordpress/","section":"Tags","summary":"","title":"Wordpress"},{"content":" I don't usually use WordPress, but when I do, I configure it with Chef. In fact, I\u0026rsquo;ve never done actual work in WordPress. Unfortunately, a WP-based project at work isn\u0026rsquo;t going so well, and we need to have team members ramp up super quick to be able to contribute. WordPress needs Apache, MySQL and PHP to run, but there\u0026rsquo;s no guarantee that a.) team members have those installed or b.) they won\u0026rsquo;t have configuration conflicts.\nThe Cookbook #Instead of telling everyone to install MAMP, I whipped up a Chef recipe and Vagrantfile that\u0026rsquo;ll get an environment running quickly. The WP-Shell Cookbook will install the necessary packages, create a virtualhost and import a database dump. The cookbook assumes that the PHP code will reside on the host machine and that the database dump will be available during the first converge.\nYou can click through the link above to check out the code, but the one thing I wanted to highlight was the DB create. I\u0026rsquo;m sure there\u0026rsquo;s a better way to bootstrap the DB (and repopulate it if the data changes). If you know of a way, please let me know.\n## import the mysql data execute \u0026#34;create-database\u0026#34; do command \u0026#34;mysql -u #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_user\u0026#39;]} -p#{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_password\u0026#39;]} -e \u0026#39;CREATE DATABASE IF NOT EXISTS #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_name\u0026#39;]}\u0026#39; \u0026amp;\u0026amp; mysql -u #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_user\u0026#39;]} -p#{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_password\u0026#39;]} #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_name\u0026#39;]} \u0026lt; #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;host-folder\u0026#39;]}/#{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db-dump-name\u0026#39;]}\u0026#34; not_if (\u0026#34;mysqlshow -u #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_user\u0026#39;]} -p#{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_password\u0026#39;]} #{node[\u0026#39;wp-shell\u0026#39;][\u0026#39;db_name\u0026#39;]} | grep wp_users\u0026#34;) end You\u0026rsquo;ll also notice that there\u0026rsquo;s a reference to 'host-folder', which creates a syntactical tight coupling to the use of Vagrant. I should probably put correcting that on the to-do list.\nThe Vagrantfile #The Vagrantfile does more than it needs to, but a vagrant up takes roughly 12 minutes - about the amount of time it would take to grab a coffee. Here are some of the key components in the Vagrantfile:\nconfig.vm.synced_folder \u0026#34;\u0026lt;path_on_host_machine_to_database_dump\u0026gt;\u0026#34;, \u0026#34;/home/vagrant/host\u0026#34; config.vm.synced_folder \u0026#34;\u0026lt;path_on_host_machine_to_wp_src\u0026gt;\u0026#34;, \u0026#34;/opt/wordpress\u0026#34; config.vm.provision :chef_solo do |chef| chef.cookbooks_path = \u0026#34;cookbooks\u0026#34; chef.add_recipe \u0026#39;wp-shell::default\u0026#39; chef.json = { \u0026#39;mysql\u0026#39; =\u0026gt; { \u0026#39;server_root_password\u0026#39; =\u0026gt; \u0026#39;rootpass\u0026#39; }, \u0026#39;wp-shell\u0026#39; =\u0026gt; { \u0026#39;server_name\u0026#39; =\u0026gt; \u0026#39;wordpress.dev\u0026#39;, \u0026#39;docroot\u0026#39; =\u0026gt; \u0026#39;/opt/wordpress\u0026#39;, \u0026#39;db_name\u0026#39; =\u0026gt; \u0026#39;wp_db_name\u0026#39; } } end It\u0026rsquo;s really quite straightforward. The two mounted paths are for the database dump and the WordPress code. On line 12 in the chef.json I show how to set some of the cookbook attributes. I had initially tried to do most of the work from within the Vagrantfile, but it fell apart when I wanted to create and enable the WP-specific virtualhost.\nIn Conclusion #Hopefully this doesn\u0026rsquo;t re-invent an existing solution - aside those like MAMP, LAMP or WAMP. Help make it better: fork it on GitHub.\n","date":"2013-07-28","permalink":"/articles/wp-shell-cookbook/","section":"Articles","summary":"","title":"WP-Shell Chef Cookbook"},{"content":"TL;DR #Why doesn\u0026rsquo;t front-end ops need to be a separate role? Because the developers building your pages know those pages best, know what tooling is needed and should be thinking about how those pages perform. How do you get those folks interested in and caring about performance and ops? Well, good luck with the ops part, but to get developers to espouse performance, create fast-feedback loops that allow performance data to be visualized. That visualization is the enzyme that allows digestion of data. Once you have that feedback loop established, keep the data fresh and the atmosphere enthusiastic.\nThe Back-story #A recent article in Smashing Magazine about front-end ops struck a chord with me. The piece, by Alex Sexton, argued in part that front-end ops should be a distinct role, differentiated from front-end development. While it was truly refreshing to see an enumeration of these tasks, I felt conflicted about the introduction of a new role. It has taken time, but in my current role, I have been able to provide an environment where the front-end developers can perform all the tasks tasks Alex mentions. I come from the development side myself, but am passionately interested in operations. I have done my best to create a work environment where we can do all the things that make us professionally satisfied, and, unsurprisingly, those same things make our products better.\nEnabling Your All-In-One #The first step to enabling this environment was to set the foundation: continuous delivery. Just to clear the air, we\u0026rsquo;re not delivering to production continuously and our process isn\u0026rsquo;t perfect, but we\u0026rsquo;ve done enough to create a feedback loop for the team. The build process takes care of everything from minification, concatenation, and running Compass to the automated functional testing grid, deploying wars, and flushing/warming caches. It has been my experience that once CI is configured, deployment mechanics fade into the background. The system is not self maintaining though. Instead of creating an explicit role to keep things running, team members who are interested in contributing self-identify and I assure time is allotted. An analogy I have frequently used is to the flywheel in a single piston engine - the engine wouldn\u0026rsquo;t be able to continue to function without the flywheel.\nAfter automating our deployment pipeline, we shifted our focus to monitoring. RUM data is one of the most enlightening pieces of monitoring information we can collect from the browser. The initial effort of implementation was low because the dozen lines of javascript to access and calculate stats from the Navigation Timing API is well defined. With basePage (responseStart to responseEnd) and frontEnd (responseEnd to loadEventStart) data* for all our pages, the team has the measurements and historical data to determine if we\u0026rsquo;re making the site faster. Once RUM monitoring logic is embedded in pages, it\u0026rsquo;s largely self-sustaining - though there is certainly more to quantifying perceived page performance.\nThis brings me to an important point about how we use monitoring data. We use graphs to visualize the performance of our most popular pages. Those graphs don\u0026rsquo;t just sit on a web page somewhere, they gets displayed on a dashboard in a highly trafficked area. The data is refreshed every minute and typically shows the past week. This helps keep the topic of performance in the weekly and often daily conversation. I won\u0026rsquo;t go into the cultural benefits of using dashboards to visualize data, but suffice it to say we\u0026rsquo;ve seen those benefits first-hand. If you want a cheap dashboard display, check out this post about building a kiosk.\nAs I alluded to above, tracking page load performance is only raw performance data and if the front-end team\u0026rsquo;s interest ended at RUM data, we would be doing ourselves a disservice. For example, what if content below the fold was loaded before the content above the fold or if the first image in the homepage carousel was the last to load? Enter WebPageTest.org . This free and open source tool gives the front-end developer all the measures they need to understand the anatomy of a page load. Especially of interest is the ability to see a filmstrip of page load progress snapshots correlated to the waterfall chart. Here\u0026rsquo;s a view of the Amazon.com homepage first view . The team saw the value of this type of analysis immediately, though I will admit that it is the least well integrated into our overall process, and therefore requires the most overt effort to sustain. We are still in the process of establishing a good cadence of WebPageTest use, but the ability to compare filmstrips from test runs on different dates allows us to benchmark the gains.\nSeparating the front-end developer from performance and optimization concerns will only do the developer - and team - a disservice. I would argue that it is leadership\u0026rsquo;s responsibility to make time for these activities. Possibly more important is to create a feeling of purpose, imperative and enjoyment for these tasks. While premature optimization is a bad thing, a well rounded developer will consider performance ramifications when laying out a page. The team will build performance into the page, much the same way quality should be built into a process.\n","date":"2013-07-01","permalink":"/articles/front-end-engineer-all-in-one/","section":"Articles","summary":"","title":"Front-End Optimization, Monitoring \u0026 Deployment Engineer All-In-One"},{"content":"Update: For the hardware-hacking enthusiasts, there\u0026rsquo;s an more in-depth post about building a XBee Kill-a-Watt transmitter (with lots of pics): XBee Kill-a-Watt Transmitter .\nXBee Kill-a-Watt Power Measurement #I first found the project on Element14\u0026rsquo;s community site , and that lead me to the original Adafruit design . The ComEd Smart Grid has been in the news lately so the idea of monitoring power consumption was intriguing. A coworker passed me a link to ElectriSense , a University of Washington project, which further fueled my interest.\nCredit: www.ladyada.net The overall system view (above) is comprised of a single receiver and several transmitters, all relying on XBee wireless communication. XBee modules have built-in A-D converters, so in this system the transmitters are programmed to sample analog values from a specific pin and broadcast those along with an identifier. If you wanted to have discrete networks of XBees, setting discrete PAN IDs would do the trick.\nHardware #I ordered the Tweet-a-Watt Starter Pack and followed the guide on how to assemble the parts. The process is pretty straight forward, but took several hours. I tend to take frequent breaks since my makeshift soldering station is uncomfortable at best, so your mileage may vary. The first piece of the system you construct is the receiver. Essentially this is a standard XBee adapter.\nA fully assembled Adafruit XBee Adapter Each XBee module, whether it\u0026rsquo;s a receiver or transmitter, needs to be programmed via the adapter and FTDI cable. I had trouble getting the XBees to consistently write firmware successfully. I did not find a true solution to this, but was successful after successive tries.\nThe Receiver - an adapter with the XBee module and FTDI cable attached The second piece is the transmitter. LadyAda\u0026rsquo;s guide has a very detailed step-by-step guide for the transmitter , though you may find that some steps vary due to revised Kill-a-Watt parts. The P3 Kill-a-Watt unit I purchased was a P4400*.1*. This revision has an updated PCB and smaller quad op-amp that you have to solder into. There is no known way to tell which version of the P4400 you\u0026rsquo;re ordering when you order online, though I would assume any models purchased at this time would be P4400.1\u0026rsquo;s.\nUpdate: I built a second transmitter and posted about it here . There are several pictures of the process.\nSoftware #The Python Serial and XBee libraries are used to pull the analog samples from the XBee receiver module. I adapted LadyAda\u0026rsquo;s wattcher.py from GitHub for my needs. I deleted the App Engine and Twitter code and added some code that pushes to Graphite.\nHere\u0026rsquo;s my adapted wattcher.py:\nLines 170-181 is the Graphite part. I\u0026rsquo;m no python expert so there could easily be a better way to achieve this. For now it works, so the code cleanup will have to wait.\nCalibration #An important part of the process is to calibrate the script to adjust for the power consumed by the XBee inside the Kill-a-Watt. To do this, run ./wattcher.py -d, which toggles on debug mode. The output will contain a line that looks like the following:\nampdata: [498, 498, 498, 498, 498, 498, 498, 498, 498, 498, 498, 498, 498, 498, 497, 498, 498, 498] From this data we can determine what the sensor vrefcalibration values should be on lines 16 to 21. My transmitter registered an average close to 492, and the calibration made a noticeable difference in the measured output.\nGraphing #I had originally wanted to use Google Powermeter to log and graph my data, but the service was taken retired in 2011. I immediately turned to Graphite. We use Graphite at work and although I\u0026rsquo;ve never had to set it up by hand, using Chef to stand up an instance is super easy (Update: I posted about my Vagrant, Graphite and StatsD setup ). Once I had the Graphite server running, the additional python was pretty easy:\nsock = socket() try: sock.connect( (DEFAULT_CARBON_SERVER, DEFAULT_CARBON_PORT) ) except: print \u0026#34;WARNING: Couldn\u0026#39;t connect to %(server)s on port %(port)d, is graphite running?\u0026#34; % { \u0026#39;server\u0026#39;:server, \u0026#39;port\u0026#39;:port } if (sensorhistory.sensornum == 1): message = \u0026#34;%s %s %s\u0026#34; % (\u0026#34;wattage.office.watts\u0026#34;, sensorhistory.avgwattover5min(), int(time.time())) sock.sendall(message + \u0026#34;\\n\u0026#34;) sock.close() The script currently pushes only the 5 minute average wattage (once every five minutes), but I intend to update that to 1 minute averages every minute. Once you accumulate some data, the graphs become quite informative.\n24 hours of power consumption for my home office My home office consumes a baseline of about 215 watts. This includes two desktops, two 24\u0026quot; monitors, a laptop (sometimes), two printers, speakers, a modem, a router, a switch and various other peripherals, so I\u0026rsquo;m actually surprised at how low the power consumption is at idle and now little that changes when they\u0026rsquo;re in use. It\u0026rsquo;ll be interesting to gather more data comparing these devices\u0026rsquo; power draw and how it compares to other areas in the house. I think my first target for comparison will be the media console.\nThe Completed System #All-in-all this was a really fun project. It\u0026rsquo;s a easily attainable mix of hardware and software hacking, and the fact that I get to graph some metrics at the end of the day is icing on the cake. I know I probably could have purchased a system that performs similar tasks, but the act of making satisfied the computer engineer in me.\nA completed transmitter If you got this far, thanks for reading!\n","date":"2013-06-02","permalink":"/articles/xbee-kill-a-watt/","section":"Articles","summary":"","title":"XBee Kill-A-Watt Power Measurement System"},{"content":"","date":null,"permalink":"/tags/kiosk/","section":"Tags","summary":"","title":"Kiosk"},{"content":" I\u0026rsquo;ve had a Raspberry Pi sitting on my desk at home for months telling me whether I had new Gmail. At $35, the Pi is cheap, but I sure wasn\u0026rsquo;t getting my money\u0026rsquo;s worth using it as a glorified dock notification. Then I saw this post from Pivotal Labs about using a Pi as a kiosk. I immediately thought of using a Pi to display Graphite graphs at work, and while it took an ashamedly long time to get moving on the project, the configuration only took an hour.\nUpdate 2015-02-25: As of the 2015-01-31 release of Raspbian (for Pi2 support), the LXDE autostart file is located at /etc/xdg/lxsession/LXDE-pi/autostart. I\u0026rsquo;ve updated the instructions to reflect this change.\nThe full setup - with the Pi exposed. Configuring the Pi to be a kiosk was pretty easy. The idea is to have the Pi boot to a full-screen browser that loads a predetermined page. Additionally, I installed VNC so that I could view the desktop remotely if necessary.\nI followed the following steps starting from a fresh Raspbian image:\nUse raspi-config to:\nenable ssh change the locale and timezone boot to desktop expand_rootfs Update and install some software:\n$ sudo apt-get update $ sudo apt-get install ttf-mscorefonts-installer unclutter x11vnc x11-xserver-utils Disable sleep so the screen stays on:\n$ sudo vi /etc/lightdm/lightdm.conf # add the following lines to the [SeatDefaults] section # don\u0026#39;t sleep the screen xserver-command=X -s 0 dpms Configure LXDE to start the Midori browser on login:\n$ sudo vi /etc/xdg/lxsession/LXDE-pi/autostart # comment everything and add the following lines @xset s off @xset -dpms @xset s noblank @midori -e Fullscreen -a http://example.com Please Note: If the LXDE-pi folder doesn\u0026rsquo;t exist on your, you may be using an earlier version of Raspbian. The correct location for the pre-2015 Raspbian is /etc/xdg/lxsession/LXDE/autostart.\nConfigure VNC to start on boot\n$ sudo curl -L -o /etc/init.d/x11vnc https://raw.githubusercontent.com/starlightmedia/bin/master/x11vnc $ sudo chmod 755 /etc/init.d/x11vnc $ sudo update-rc.d x11vnc defaults Chromium #If you prefer to use Chrome as the kiosk\u0026rsquo;s browser, you can do this:\n$ sudo apt-get install chromium and:\n$ sudo vi /etc/xdg/lxsession/LXDE-pi/autostart # replace the midori line with the following @chromium --kiosk --disable-session-crashed-bubble --disable-restore-background-contents --disable-new-tab-first-run --disable-restore-session-state http://example.com Refreshes #The Graphite dashboard pictured at the top of this article seemed to have a memory leak because both Midori and Chrome would crash after running the dashboard for about 20 hours. Instead of trying to fix Graphite I treated the symptoms. Xdotool is one of the handy utilities written by the awesome Jordan Sissel and allows you to simulate keyboard and mouse input.\nsudo apt-get install xdotool and add the following to crontab (you\u0026rsquo;ll need to make sure the name matches the browser application you\u0026rsquo;re running):\n0 */6 * * * DISPLAY=:0 xdotool search --name chromium windowactivate --sync key F5 That should be all you need to do to get the Pi configured. I use RealVNC to connect to the Pi.\nIf I\u0026rsquo;ve left something out, please leave a comment and let me know.\n","date":"2013-05-31","permalink":"/articles/pi-dashboard-kiosk/","section":"Articles","summary":"","title":"Raspberry Pi Dashboard Kiosk"},{"content":"Go has been a popular language for a while and the chatter has gotten to the point where I needed to give it a spin to see what all the fuss is about. The night before a flight to SFO, I installed the Go distro and found the wiki tutorial . I also made sure to get the Go package installed for Sublime so I wasn\u0026rsquo;t stuck in black and white.\nFirst, I\u0026rsquo;ll start with the disclaimer that I\u0026rsquo;ve been using higher level languages like Groovy and Ruby recently, so I haven\u0026rsquo;t been writing code that gets quite so explicit. The first thing that stuck me was the use of := for assignments. For example:\nfilename := title + \u0026#34;.txt\u0026#34; Turns out that\u0026rsquo;s just the shorthand for var filename = title + \u0026quot;.txt\u0026quot;, which is actually receiving it\u0026rsquo;s type from the assignment in the expression.\nThe next several steps in the tutorial were pretty self explanatory. The syntax for the way functions can return multiple variables was intuitive, and I really liked how the compiler complains when you don\u0026rsquo;t use imported packages. The html/template package\u0026rsquo;s Must function was also a handy way of ensuring that expected resources are present at startup.\nThe last language feature that the wiki tutorial covered was function literals and closures. Groovy is littered with closures, but I have most frequently used them to perform operations on items in a collection:\n// data is: [[name:red, hue:#f00], [name:green, hue:#0f0], [name:blue, hue:#00f]] def colors = data.collect { [\u0026#34;$it.name\u0026#34;: it.hue] } println colors // [[red:#f00], [green:#0f0], [blue:#00f]] The Go closure from the wiki does something a bit more interesting. It wraps a controller action (also a function) with the functionality to validate that the requested resource matches a validator - in this case defined elsewhere as a regex:\nfunc makeHandler(fn func (http.ResponseWriter, *http.Request, string)) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { // do the work title := r.URL.Path[lenPath:] if !titleValidator.MatchString(title) { http.NotFound(w, r) return } fn(w, r, title) } } Pretty cool stuff. After the plane ride I did the other tasks suggested by the tutorial and added Twitter Bootstrap to make it prettier. I also watched a video about the origins of Go and why Google invented it. The video helped make sense of some of the languages quirks.\nI don\u0026rsquo;t yet see a clear fit for Go in any of my current work. Our core deliverables are highly branded websites, and Go certainly wasn\u0026rsquo;t created with the goal of assisting in that development process. In my limited development capacity, I\u0026rsquo;ve been creating better observability inside the application or providing the tools to create meaningful visualizations with all of our metrics. We have the intention of collection RUM data in the future and maybe Go would be a good way to collect that data. Additionally, projects like gor catch my eye because it would fill a gap we\u0026rsquo;re looking to fill in the future.\n","date":"2013-05-30","permalink":"/articles/trying-go/","section":"Articles","summary":"","title":"Trying out Go"},{"content":" As you may have guessed from a previous post , I prefer Java Config. At times in the past I\u0026rsquo;ve been forced to go with what\u0026rsquo;s currently supported instead of spending the time necessary to add the feature I\u0026rsquo;d like to have. With the Spring Grabbag project, I have that time.\nThe current mybatis-spring project (v1.1.1) only has support for XML-defined config. Some searching lead me to java/xml config hybrids, like the one described here , but that seemed too much like the compromise I\u0026rsquo;d been forced to make in the past. Some more targeted searching lead me to this issue on the MyBatis issue tracker, which in turn lead me to a more technically robust ticket on SpringSource\u0026rsquo;s Jira instance.\nAt this point I knew:\nthere was no code yet-written to satisfy my predicament that the SpringSource folks expected the MyBatis-Spring code to follow the established @Enable pattern that Chris Beams had very graciously written code that reproduced the situation and committed it to the spring-framework-issues repo Now all I had to do was learn the Spring bean lifecycle and write the solution. While the documentation is great, it\u0026rsquo;s also a bit dense. I chose to read up on EnableWebMvc , ComponentScanAnnotationParser , AspectJAutoProxyRegistrar , and BatchConfigurationSelector and more specifically how configurers, adapters and registrars are used in the context of Spring annotations. Given that the MapperScanner is registering new beans to the application context, adding a registrar made most sense. I had been in contact with Eduardo Macarron , the maintainer of the MyBatis-Spring project, and he altered the code as necessary to be accepted into the code base.\nIt feels good to be able to @MapperScan my DataConfig and have it work like you\u0026rsquo;d hope and expect. It feels great to meaningfully contribute back to a project I\u0026rsquo;ve relied on so much in the past.\n","date":"2013-01-19","permalink":"/articles/mybatis-spring-java-config-contribution/","section":"Articles","summary":"","title":"MyBatis Spring Java Config Contribution"},{"content":" Photo credit Grant Hutchinson I\u0026rsquo;ve been a big fan of Spring since the summer of 2009, when a teammate began to introduce Spring MVC Java Config into an old code base we were working on. It was the first time I\u0026rsquo;d used Spring in production, and it didn\u0026rsquo;t take long for me to become quite fond Spring\u0026rsquo;s modularity and overlook it\u0026rsquo;s flaws. Over the next few years that team converted the bulk of that website into a healthy Spring 3.0 MVC implementation.\nFast forward a couple years, and I don\u0026rsquo;t have any Spring-based stacks running in production. At work we\u0026rsquo;ve created a handful of Spring apps that use various ORMS, schedulers, generate reports, have various config styles, etc., but I haven\u0026rsquo;t been able to closely influence the direction of these apps. Spring Grabbag is an attempt to create a place where I can go to reference all that I\u0026rsquo;ve done with Spring. I very much doubt that I\u0026rsquo;ll be able to add even half of what I\u0026rsquo;d like to, but it\u0026rsquo;ll be a cathartic experience nonetheless.\nSince I started writing it over the annual holiday family visit, a digital cookbook came to mind. The concept of a cookbook should provide enough of the structural facets to prove interesting. We\u0026rsquo;ll see how it goes. If you\u0026rsquo;re wondering what\u0026rsquo;s on the to-do list, have a look ","date":"2013-01-06","permalink":"/articles/spring-grabbag/","section":"Articles","summary":"","title":"Spring Grabbag"},{"content":"Last week I was catching up on some DevOpsCafe episodes and heard John mention a great presentation on AMPQ. It reminded me of a session I\u0026rsquo;d attended at SpringOne a year earlier, but couldn\u0026rsquo;t remember who had presented. The session was about polyglot messaging with AMQP, and while I didn\u0026rsquo;t fully appreciate it at SpringOne, the subject now perked my attention. After some searching, I connected that dots that Rob Harrop, who John had mentioned, was also the presenter at SpringOne. I watched the more recent presentation on InfoQ and was inspired to send some messages of my own.\nThe demo I created doesn\u0026rsquo;t do a whole lot beyond what Rob did in his presentation, but it sufficed for a presentation I gave at work to illustrate the concept. Unfortunately I don\u0026rsquo;t see any immediate application for AMQP at work, but now that I\u0026rsquo;ve demoed some code, hopefully the barrier to build more has been broken.\n","date":"2012-10-27","permalink":"/articles/amqp-demo/","section":"Articles","summary":"","title":"AMQP Demo"},{"content":"","date":null,"permalink":"/tags/rabbitmq/","section":"Tags","summary":"","title":"Rabbitmq"}]