You CAN measure the return on training investment: a case study

ROI: it’s an acronym that causes gastric distress for many performance leaders in industry and government. When everyone has to do more with less, it seems like the right thing to do – to measure how - and if - our training investments are actually working.

And of course, it is the right thing to do. So why does it seem to be so doggone hard?

Well, it doesn’t have to be.

Using a simple case study as an example, we’ll walk through how to measure the value added to an organization following a large software training initiative. You can use this model as a template for a variety of measurement needs.

And while most ‘creative training types’ cringe at higher math, me included, not to worry. This model is very much paint-by-numbers.

There are lots of ways to measure training effectiveness. That’s mostly because we can define “measure” and “effectiveness” in different ways. But that’s a different conversation for a later time.

Here are two good articles on how to measure training ROI:

Keep your training program funded

How to measure return on training investment

So why do you want to evaluate training effectiveness, anyway? Because it can and does:

• Improve training quality: evaluation forces us to think about improvement - and we are more likely to make improvements as a result

• Increase learning: when we determine how much was learned we can make improvements for more and better learning

• Identify roadblocks to learning and job-skills transfer, and eliminate those roadblocks, which improves proficiency and productivity; which

• Help the organization achieve its mission – and improve the bottom line

Also, it’s the right thing to do. It demonstrates the value of learning in the organization.

Here’s the model in the form of a case study:

Sue is the training manager of an organization with approximately 4,000 employees. She and her team of three conducted a series of software classes, both instructor-led and online. The software included both word processing and spreadsheet programs which were new to all employees.

Before the classes were rolled out, Sue decided to measure the return on this training investment and report the results back to leadership. This was her golden opportunity to demonstrate the value of her training unit. Leadership didn’t ask for this, and they kinda shrugged when she told them she wanted to do it. The shrug was all the permission Sue needed. She was a corporate badass warrior who knew what needed to be done.

Her first step was to define how and what she would measure. She wanted to see what kind of productivity increases resulted from the computer skills training, and defined this as “value added” to the organization.

Her definition of “value added”: The dollar value of performance improvement associated with the tasks performed on the job using the skills learned in training.

For the mathematically oriented, it looks like this:

Value added = (S x T)(P2 – P1)

S = Annual salary

T = % of time spent on specific tasks

P2 = Predicted productivity AFTER training

P1= Productivity BEFORE training

For the mathematically-challenged, it looks like this:

Salary times Time (percent of time spent on those specific tasks) multiplied by the % of productivity change.

Here’s how Sue’s team collected the data:

These questions were added to the post-training ‘smiles sheet’ that participants completed after class (the Level 1 evaluation):

1: What percent of your total work time will you spend on tasks that require the knowledge/skills provided in this course?

2: Rate your productivity before training on the tasks that require the knowledge/skills provided in this course (on a scale of 0% to 100%)

3: Predict your productivity rate after training on the job tasks that require the knowledge/skills provided in this course (on a scale of 0% to 100%)

4: Annual salary: ranges were offered with several choices to select. (The evaluations were anonymous, of course.)

Let’s plug in the numbers for a typical participant:

The formula, using a salary of $40,000/year as an example, where this employee estimated he spends 10% of his work day on tasks using the skills learned:

(Salary x Time)(Productivity AFTER x Productivity BEFORE)

$40k x 10% = $40,000

Estimated 80% productive AFTER

Estimated 40% productive BEFORE

80%- 40% = 40% increase – So…

$4,000 x 40% = $1600 value added

Next, Sue knew that she needed to validate this data with some form of reality. How did the training help people on the job?

She asked class participants for their productivity estimates approximately 60 days LATER, after the enthusiasm of the class had worn off. This gave employees time to actually use the skills they learned in the class.

The team sent an email with the same four questions, along with a brief explanation about the evaluation follow-up. The team randomly chose 4 participants from each of the 10 classes they taught, and the same number for the online courses.

1. On a scale of 0% to 100%, how much time do you spend working on tasks that require the skills you learned in the such and such training?

2. On a scale of 0% to 100%, what was your productivity rate on those tasks before you went to the class?

3. What was your productivity rate on those tasks before you attended the training?

Now when they plugged in the numbers, they had a more accurate assessment of how the skills were being used and could put this data into qualitative terms – dollars and productivity increases.

The punch line to this case study is that it really happened. In state government!

Most important of all, our leadership and employees recognized that we were a learning organization, and they became more invested and enthusiastic in its success.

The results were surprising. We used a control group of 20 software classes, and determined that the value added for the entire series of 192 classes delivered that year returned a value of over 5 million dollars! The average value added per employee was $3,300.

Our annual training budget was increased significantly the following year. (Our team leader was Roy, not Sue, and he was more mentor than badass.)

Most important of all, our leadership and employees recognized that we were a learning organization, and they became more invested and enthusiastic in its success.  Did I mention this happened in a state government agency, one that had just undergone a disruptive merger with another agency?

If it can happen there, it can happen with your organization, too. And you, an evaluation warrior, can lead the way. Good luck!

How do I know if training is the solution?

Sometimes training is just not the solution. Or, it may be part of the solution, or a follow-up to the best solution. It really just depends. Like life in general, the answer is not always black-and-white.

One thing is for sure, though: needs assessment is the first step in the
process of any training design. Or, preparing any presentation on any topic. Or even deciding how to pitch an idea at your next professional association meeting. It’s what you do instinctively when you’re assigned  a new project.

You may not call it a ‘needs assessment’, but you go through the same motions regardless. When you wonder:

• “what is the objective of this assignment?”

• “what outcomes are expected?”

• “how will behaviors change as a result?”, and

• “why is this an assignment at all?” then you are asking the right needs assessment questions to answer before you can dive into your new project.

So how do you know for sure that the solution to the problem is, in fact, training? If an employee cannot perform a task, even if you put a gun to their head (apologies for this violent example), then training is the solution. When the employee simply does not know HOW to do that task - training is the solution.

If, however, she can do the task under the right conditions, but those conditions don’t exist very often, then training is not going to solve the entire problem. The problem might be a workplace design issue, a crummy supervisor, or an organizational process that needs tweaking.

Needs assessments don’t have to be long, drawn out, laborious projects. Not many organizations do them in a textbook way, and many recoil at the mere mention of the prospect. In my thirty years of work in the field of training and development, I can count on one finger how many organizational needs assessments were requested by senior leadership. (It’s important to distinguish here that I’m not referring to a training needs assessment, where training is assumed to be the solution, and the goal is to assess what types of training are needed.)

Sometimes an organization will want a class or online course to teach employees how to do X, because X is required for some good reason. Understanding sexual harassment policies, fair hiring practices, and cyber security are a few examples of classes needed to ‘check a box’. And in those
cases, those boxes are important, so no further questions need to be asked. Just get the training out there and check that box.

In other cases, a series of questions may be all that’s needed for a quick and effective needs assessment. For example, let’s say that your Chief Learning Officer was asked by the CEO to develop an online course on how to transfer customer service calls to the correct service representatives. The error rate is too high, calls are going to the wrong departments, thereby wasting a lot of time and resources re-transferring calls. Everyone is frustrated. The training needs to be quick and efficient. So, the CEO asks, how long will it take to develop and put a course out there?

A few obvious questions spring to mind, and absolutely, this is a quick-and-dirty needs assessment:

1.    What is happening now with the customer calls? Can I talk to a few customer service reps to get an idea of their specific processes? Is there a ‘model employee’ who does it right each time?

2.    What specifically is causing the frustration for employees? For customers?

3.    What skills are needed to transfer these calls perfectly? Are these skills that the customer service reps already have?

4.    What processes are in place to support these call transfers?

The answers to these questions are your initial needs assessment. By drilling into these answers and perhaps doing a cursory job-task analysis, you can determine whether training will solve this problem.

You decide to take an hour and go to the customer service desk to ask a few service reps some of these questions. They are more than happy to share their frustrations with the new automated phone system. Customers don’t understand the company “lingo” on the call message, so they press the wrong option. “Option 4 for Service Questions and Billing” is confusing. It isn’t the customer’s
fault – they just don’t know what that means. All they hear is the word “billing”, which makes total sense because we hear and comprehend the last thing that was said.

The most important part of the process is to trust your gut.

Had anyone asked the service reps before why the new system wasn’t working? Well, no, and they were not going to tell the manager. Retaliation for this type of “complaining” was swift and harsh. Oh. Well, then. The easy fix is to change the voice system message and wait for the error rates to improve. Addressing the issue with the retaliatory and problematic manager is a bonus.

The most important part of the process is to trust your gut. And trust the process.

When you suspect a solution to the problem, even if it may be challenging to implement or unpopular, go ahead and trust it. Then validate it to be sure. And then communicate it. To not do so is a disservice to you, employees, customers, and the entire organization.

And from this starting point, everything else will fall into place.

Nature and Creativity: 5 Practical ways to find inspiration and solutions

One sunny day last week I drove to the grocery store. This kind of errand used to be a chore. Now it has become a big deal during this pandemic, an outing that I look forward to each week. Like so many millions of people, I have to be careful with vulnerable family members who simply cannot get Covid19.

So, on my long drive through the countryside, I noticed the cows and deer grazing in fields, neighbors bailing hay, geese swimming in ponds, and the Blue Ridge mountains of Virginia holding up a clear and cloudless sky.

And then it hit me. The solution to a work problem I had been pondering for days. While thinking of nothing in particular, the solution came to me – and it was a good and creative one - if I do say so myself.

Lately my pandemic lockdown thoughts have turned to how nature can help get that creative mojo flowing again.

I know this happens to everyone, and I started to wonder why. Much has been written about how nature inspires creativity and problem solving.  Studies have proven that being in nature reduces stress, which is especially helpful these days.

Five proven ways spending time in nature inspires creativity

Ten creative benefits of spending time in nature

Lately my pandemic lockdown thoughts have turned to how nature can help get that creative mojo flowing again. Before the pandemic, I used to just change my setting. I’d go to the library for a few hours after attending an in-person meeting. Or I might spend an hour in a coffee shop, working on my laptop while people-watching and sipping a hazelnut decaf. Ah, those were the days.

I live in the middle of the woods, in the middle of nowhere. These changes of place were just what I needed.  Without the option of working in another location, how can you re-set that creative spark?

Well there are some simple ways, if you go about it with planning and intention. This is not scientific, although it has worked well this past month or so. I would like to make it a lifelong habit, with tweaks along the way.

Step One: find a nearby spot in nature that soothes your soul. It could be your backyard or a park in your neighborhood.  Anywhere outdoors and peaceful will do. With a creative goal for the day (or week) in mind, pick a good time when your stress level is kind of low, and then go to that place.

Step Two: while there, focus your mind on your senses and ‘exercise’ each one. Walk or sit on a bench and observe, really see your surroundings. Listen. Smell. Touch. Most importantly, breathe deeply. Take a video or a picture with your phone. This will be useful later.

Step Three: before you leave, find a souvenir. Something simple and easy to put on your desk: a pinecone, a rock, a wildflower, a seashell. Make it a souvenir that represents the problem you want to solve. If you need to come up with a “thorny” design solution at work, one you don’t have experience with, a pine needle might represent that problem, for example. Your souvenir will serve to remind you of your creative goal for that day. The picture or video you took could serve as the souvenir, too.

Step Four: as your souvenir pine needle sits on your desk during the week, each time you see it think for just a moment about your solution. Don’t focus on it for long, just a few seconds. That’s all you need to get the creative coffee grounds percolating.

Step Five: set a date to do this regularly - at least once a week is ideal. Put it on your calendar. Set a new creative goal for each sojourn and keep it loose. No pressure, just be open to whatever might happen that day.

If you miss going to your nature spot one week, that’s okay, life happens. Instead, use an outdoor chore and set your mind to creative energy and solutions. If you need to wash your car, weed the garden, or walk the dog, decide what you will focus your thoughts and creativity on during the entire chore.

Like meditation, when your thoughts stray, just bring them back to the task at hand. The solution you’re seeking will likely appear later, when the chore is done and you’re thinking of something else. But the seed was planted with conscious intention.  And all seeds grow – and bloom - while you are doing something else.

With some practice and dedicated time, solutions and creative inspiration becomes, well, just a walk in the park.