In 2011, DoSomething.org, a non-profit organization, posted a video on YouTube featuring celebrities asking young people to donate used sports equipment to youth in need. That video was viewed by 1.5 million people. Success, you might think? Well, only eight viewers signed up to donate. Of those eight, none actually donated.
What happened? How could such a huge number of views not produce any donations? “We were concerned with the wrong metric,” writes Jeff Bladt, Director of Data Products & Analytics at DoSomething.org, in his blog. “In the case above, success meant donations, not video views. As we learned, there is a difference between numbers and numbers that matter. This is what separates data from metrics.”
Track metrics, not data. Tracking the wrong data left DoSomething.org with no donations. Likewise, a bank’s operational intelligence (OI) solution collecting the wrong data can produce ineffective and, worse, potentially misleading actionable information. There are various ways to pick the correct metrics for a bank’s OI purposes. Let’s look at one method.
Look at an individual’s performance objectives. The process of defining meaningful OI metrics starts with an individual. Each OI dashboard has been designed for one person (or a category of persons). That individual is expected to achieve measurable operational performance objectives. Let’s say that the individual this dashboard has been designed for is named Edward. He is the supervisor of a back-office operation for high-value payments. Edward’s operational objectives are to never miss a cut-off, always meet the Fed’s deadlines, and achieve higher satisfaction levels on customer surveys. These objectives are mapped to the bank’s customer commitments.
Map the person to the bank’s commitments, obligations, and deadlines. In our example, Edward’s objectives map to the bank’s:
- Customer commitments (e.g., process all payments by cut-off)
- Regulatory compliance obligations (e.g., meet the Fed’s deadlines)
- Current strategic initiatives (e.g., improve customer experience)
Once that has been mapped, the OI metrics need to align with these objectives.
Align the metrics. Select the metrics that align with Edward’s objectives and the bank’s goals. Let’s take the example of Edward’s objective to never miss a cut-off. The goal is to proactively identify and resolve process issues prior to putting such a cut-off at risk. Therefore the OI measurements and metrics must create the necessary situational awareness to alert Edward when a situation requires his immediate attention.
An example might be the measurement of abnormalities that could lead to putting a payments cut-off at risk. In this case, OI would measure payments stuck in a step for too long. And it would measure when an abnormally high number of payments fail their STP route and go into a manual repair queue. Other things that would be measured are files or messages not received on time from a customer. Any of those situations will alert Edward and trigger him to take proactive actions leading to the avoidance of missing the cut-off. In addition to the operational objectives, there’s one more measurement to take.
Measure the OI solution to ensure that the metrics matter. Do this by measuring the data before and after the OI solution has been in production. In our example of Edward, there could be two metrics for measuring the “before and after” data. One metric could be the number of payments that missed their cut-off as a percentage of the total payments processed daily. The second metric would be the value (i.e., the dollar amount) of missed payments as a percentage of the total value processed daily.
At first glance, getting 1.5 million YouTube views seemed like success for an organization. But it was a failure because their goal was donations, not views. Likewise, it’s important for a bank to look at its goals, and to map its OI solution to track the results that will achieve those goals. Remember:
- Look at the operational objectives of the person the OI dashboard was designed for
- Map those objectives to the bank’s commitments, obligations, and deadlines
- Align the metrics
When you go through those three steps, you’ll find the measurements and metrics that matter.
Source: DoSomething.org blog post Know the Difference Between Your Data and Your Metrics