DeveloperWeek 2020

Most organizations adopting an agile approach to development and testing face a common challenge: how do I increase development velocity without allowing critical errors to slip past into production? The expression “separate the signal from the noise” comes up often, but when our systems continue to get louder and louder – more errors, more alerts, more logs to sift through – finding the one anomaly that actually risks introducing a SEV1 issue can be a seemingly impossible feat.

This was a problem my team encountered when monitoring our own systems. We were capturing tons of data about billions of events happening every day, without a quantifiable way of knowing what to fix first and how reliable our releases and applications actually were. So our R&D team set out to devise a formula for not only clearly defining what constitutes an anomaly in our code, but also for helping to prioritize the issues that actually matter and actually blocking them from making it to production in the first place.

This session will walk attendees through our open source reliability scoring system in Grafana and a variety of CI/CD tools that provides DevOps teams with a formulaic approach to prioritizing anomalies and understanding the stability of their releases – even before they go into production.

The formula scores releases for stability and safety based on if it:
-- Introduces new errors
-- Makes existing errors happen at a higher rate (i.e. rate increase)
-- Introduces slowdowns into the environment

We'll discuss how to do the below process and include a real-world example of the process being applied in practice:
-- Detect all errors and slowdowns
-- Classify each event
-- Prioritize by severity
-- Score the release
-- Block release
-- Visualize the data