DeveloperWeek 2020

Over the last 12 years, Credit Karma has grown to over 85 million members, added many new products, and expanded to an international user-base. Our huge member base offers a tremendous opportunity to champion financial progress for everyone--but only if we can find ways to reach out to them at scale.

Over the course of our growth, we’ve had to maintain a system with a non-linear PHP runtime, with very data-intensive SQL queries, running on scores of physical servers, and taking 20+ hours every day to send emails at scale; it took two weeks to email all of our users, and our operational overhead costs were unsustainable.

The first part of this talk details how we optimized the algorithm that was at the heart of this process. Instead of evaluating every user for eligibility for certain criteria, we now evaluate all users that qualify for the given criteria--resulting in a linear algorithm. The new algorithm is implemented as a set of Apache Beam transforms and deployed as a Dataflow application with BigQuery as its data source. Multi-threading our Dataflow application further parallelized our pipeline, and increased the speed of our eligibility selection process by an order of magnitude, while substantially reducing the operational overhead and the number of site incidents.

The second part of this talk describes how we scaled our sending process. Our downstream micro-services initially sent hundreds of thousands of emails/hour. But on peak days, we wanted to send several orders of magnitude more emails, which would take two full days! When we dug into the metrics and quotas and limits, we realized we had neglected the latencies of the downstream services. By horizontally scaling them and adding more consumers, we were able to scale the delivery rate to eight times with even more room to grow.

Find out how our new modular platform gives us a 5x speedup delivering emails, with dramatically reduced operational overhead, easier tuning, and fewer site incidents.