Sunday, March 04, 2007

Call Center : crunching some numbers

Alternate Title : Tech Support with an Axe to Grind

Way back in October last year I ran a post about McAfee, Norton, and Microsoft's tift over Vista Security. I followed that up with an expanded article containing the original content, but filling in the holes and gaps. This post is built in somewhat the same manner. Silly Ole Bear ran a post on MepisLovers about a Script Reader from his local ISP. Rather than just cold dumping the post into Blogger, like the Operating System Security thread, this is going to be a bit of a remix, threshing out points within the posts I made.


The Setup

SilverBear started what would become the thread at this post:
!!!

And no doubt they are starting to get the same uncomfortable feeling that a novice exorcist gets when the demon starts laughing at him halfway through the ritual!

[Just click on the START button, type what I tell you and be GONE!!!! BE GONE!!!! Away with you, foul Customer of Darkness. . . oh crap. . . it isn't working. Maybe if I gargle with the holy water. . . ?]

From the other side of the spectrum (the support side), that can be a nightmare. The first part of any troubleshooting I would do with a customer who couldn't connect is an Ipconfig, which needs the DOS box to open up. Many Original Equipment Manufactures (OEM's) like Dell, HP, or Gateway, would remove the MSDOS, DOS, or RUN lines from the Windows Start menu. It was not uncommon either to find a Windows Xp or Windows 2000 machine modified to crash out either cmd.exe, or command.exe, most often by limiting usage rights to one of the executable file names.

Under Windows XP it also was not uncommon for many users to have the Control Panel's category view set to "Expand" through the Start Menu. The result to the end user is that their accounts could be locked out of tools needed to troubleshoot their hardware or software problems, with no easy recovery method, unless the tech on the line knew how to navigate through the Start Menu Properties options by heart.

Another common change is that the Address Bar is removed from the C:\ drive Internet Explorer View, and is locked. This is a large problem when the customer only has Internet Explorer for a browser, and IE is hijacked. Knowing how to get the Address Bar under the File Browser view is useful, and may re-enable temporary access.

A lot of Windows users go on and on about how the Windows "Desktop" is unified, about how it's always "the same," and so forth. Bull Manure. There is a difference between "Customizable" and "Driving you bonkers trying to unlock stuff."

However, SilverBear is correct in that the techs are reading from a script. When I left Sitel one of the complaints I leveled against Sitel, Cox Communications, and Suddenlink Communications, is that tech support wasn't tech support. "Fixing" the problem on a call, correctly diagnosing the problem, or even determining who the customer could contact or go to for assistance, was less than 10% of the quality grade. Following the "Script" to the letter counted for far more of the "quality grade." If you didn't solve the problem at all, even if you didn't know what the problem was, you could still get a 90% plus on a quality grading. From my perspective, asking Quality Representatives what I should have done, or could have done, or what I should do on a call met with blank stares. QA personal were not interested in the customer's problem, or how the customers quality of service was presented. All QA expressed interest in was following the script.

Speaking for myself, I came from the school of training where if you were tech support, you were required to Duplicate, Replicate, and Simulate every single bloody problem the customer could have. And you, personally, had to figure out how to get out of it. If a client could bring up a C++ debugging error while running City Of Heroes under Cedega, it was your job to figure out how that error was produced to begin with, then figure out what steps needed to be taken to resolve the problem.

From my viewpoint, I consider what Sitel is/was doing to be dead wrong, and that viewpoint was one of the influences that affected my own decision to leave. The behavioral pattern is not limited to Sitel, Cox, and Suddenlink Communications. There are several other Internet providers, Computer Manufacturers, and Software Vendors who feel that Tech Support is best serviced by people reading from scripts with no practical computing experience, nor any practical training for support. The good news is, since they (the vendors) likely will never learn, that increases my chance of earning a paycheck providing the support that they (the vendors) are not willing to provide.

Is this Cost Effective?

SilverBear responded again to one of the points, and asked a question that I ducked in the thread (mainly because I'm pretty sure my first response was going to drive readers to tears)

I've looked at some of the stuff on your site, and it's very revealing to read some of your experience there. Wouldn't you summarize it by saying that it's just more "cost effective" for these companies to hire people who can read scripts that cover-- maybe, what?-- 80% of the problems, than to put techs who they'd have to pay more all the time to cover that small percentage of problems that fall through the cracks in the script ?

Unlike the MS or Mac or Linux choices that are available if one looks, in some areas if you want broadband internet, you get one choice, or if you're lucky, 2. Where I am, it's DSL through our local phone company or dial-up. Period
Fortunately for us, it's a small, locally-owned company that does try to do right by you if you have a problem.

In the village about 4 miles away you also can get RoadRunner through Time-Warner Cable service, as of late 2005.

But what would cause the average big boys to change their approach, absent competition?

Unfortunately, the practice is not cost effective, not by any stretch of any imagination, to have 80% of the problems covered by people reading from a script.

The Sitel Break Down

Back in 2003 and 2004 our Sitel (Sitel Augusta), was reported to be one of the highest earning Sitel Units. According to internal reports, this was supposedly entirely from the "Cox Project." Some of the terminology included in the reports were things like "first call resolution" or "Correct Identification of Issues" Things were so good that Cox Communications had signed an unprecedented time of service contract. When I started at Sitel on the Cox Project we had something else nobody in Sitel had. We had Job Security. We knew that 5 years down the line we were still going to be getting Cox Communications Paychecks. We were not just the best performing Sitel, we were the best performing High Speed Internet Support in the business. Thanks to technicians at Sitel Augusta, Cox Communications was getting praised for it's excellent support and knowledgeable technicians.

During 2005-2006, the reports changed as the "technicians" left. As I saw it, our Upper Management didn't just waffle, the upper management completely lost it. One of the Senior Managers told me directly that Sitel's goal was to have Sitel Augusta become the Premier Video support center for Cox Communications. Now, understand that the majority of people working under the Cox Project had answered ads looking for High Speed Internet and Computer Service Technicians. We were not Video Technicians. Most of the training that we had personally received on video consisted of "Send a hit / Roll a truck."

Okay, fine. I do know the difference between a Composite, Component, Svideo, DVI, HDMI, and DisplayPort plug. I know how to re-wire a Sun Microsystems work station monitor to use a DSUB plugin. Attempting to explain the difference to a customer over the phone, who can't see what they are doing is hard. I'm not saying that I was not capable of supporting Video Calls...

I'm saying that is not what I was hired to do. That is not what we were official trained to do.

As I saw it, the Sitel Management chased after "small job" after "small job." Remote Dial Access Support, Voice over IP support, Video on Demand Support, Billing, and so on. Our management didn't have a clue what it wanted, or why it wanted it, and the result showed. The technicians who had only a year or two before been patting themselves on the back, were leaving. Call Volume Queues climbed as employees bailed ship. When I left we were doing so badly that our contract was sold to SuddenLink communications. Our job security had vaporized because some Accountant had decided to be cost effective instead of Job Effective. Whoever was running the show apparently decided that it would be better to have people sitting in chairs reading from scripts, than properly training technicians to answer calls and fix problems.

There are several fundamental problems with the "Script" system that make it a money loser, and ultimately cost ineffective.

Problem #1 with Script System: Consumer Interaction

The first is consumer interaction, or how many calls it takes to solve a problem. Each support call is supposed to take about 10 minutes, presumably accounting for everything that could go wrong during the call. If the employee taking the call is reading from a script and has no troubleshooting experience, nor any training on how to proceed with troubleshooting, there are three potential scenarios.

  1. The first is that the Employee will attempt to fix the problem. It is possible for the Employee to waste valuable minutes of the call tracking down various issues in the manual or from a manager. This interrupts the manager's time on their own tasks, and potentially increases the length of time till the call is completed.
  2. The second is that the Employee will not fix the problem, or give the customer inaccurate information about the problem, resulting in the customer having to call back.
  3. The third is that the Employee will give the customer bad information, potentially damaging the product being called about.

The practical result is that a customer can call back 2, 3, 4, 5, 6, or more times. Presuming 10 minutes per call, the employees are now being paid to troubleshoot the same problem multiple times. If a customer has to call back 4 times in one day, there is now a total of 40 minutes spent on what was budgeted to be a 10 minute solution.

From the customers perspective, the result is frustrating. Their opinion of the company is worsened, and there is an increased chance that the customer will take his business elsewhere.

Problem #2 with Script System: Call Volume

The second fundamental problem is Call Volume, or the number of calls that are taken in a day. Call Volume directly relates to Consumer Interaction. Let us say that 20% of the calls are now callbacks. A Customer has to call in twice for the same problem. The number of calls being taken for problems has now been artificially inflated.

Lets extrapolate the effect of the call backs out.

We have 10 Employees who are expected to take 6 calls an hour, at 10 minutes per call. That is 60 problems in one hour. If 20% of these problems are call-backs, the number of calls is increased to 72 per hour. Now, if all the Employees can handle is 60 calls each hour, where are these extra 12 calls going to go? What happens in the next hour when you have another 12 call-backs, which increases the number to 24 call-backs.

The problem keeps stacking exponentially. Now you have a queue stacking. The only thing you can do is continue to take calls until no more people are calling (clearing the queue). The people in queue are now having to wait on hold.

What does that do to the average consumer? Haven't we all seen the comics about having to wait on hold? Haven't some of us done that? Trust me, getting blasted by somebody whose waited for over 2 hours with Cox Communications hold music ain't fun.

Financially, as an Employer, running a tech support crew that is not tech support poses a huge financial burden. Even with a "hoped" for 20% callback rate or "higher problem" rate, a script based staff that focus's on soft skills and not troubleshooting skills will be spending more time "on the phone" having to re-take problems they do not know the answers to. From the customers side they get the impression the technical support is useless, and then start bad mouthing the company, that whole "word of mouth" thing.


The Number Crunching


From my perspective, having been both a caller, and a technician, the Accountants in charge of funding are extremely short sighted. To an accountant it makes more sense to higher 192 Employees with no Technical experience and training, and have them take calls at 6 calls per hour, 10 minutes per call, and only pay them $8 an hour for 40 hours each week. Figuring that they would work in 3 shifts with 8 hours per shift, there would be a maximum of 64 employees taking calls at any one time. (as 40 hours with 8 hour shifts only accounts for 5 days, not 7).
Number Crunching :: The Cost Effective Approach

64*6 = 384 :: So about 384 calls each hour. Figuring a constant call volume over 24 hours, the daily volume should be 9216 calls.

So, 192 people * 8 dollars = $1536 :: $1536 dollars * 40 hours = $61440 :

The payroll would run $61440 for each employee to work 40 hours a week. $61440 / 5 days = $12288.

The expenditure figuring a 5 day work week would be $12288. $12288 / 9216 ~ $1.33 :: so each call is worth about $1.33

Now, lets screw the numbers all up. Lets say that 20% of the 9216 daily callers have to call back in. That is about 1843 (1843.2) extra calls coming back in.

If the company that is paying for the calls pays for the extra 1843 calls, they'll pay $1.33 per call. That would be about $2451.20 extra from the company, per day. Multiply that times 5, and the additional 5 day work week cost is about $12256.

If the contractor taking the calls takes the hit out of their own pocket for the call backs, say, they pay time and half to handle the calls... lets run those numbers.

8 / 2 = 4 :: 8+4 = 12. So, $12 per hour. 1843 calls / 24 hours ~ 77 calls an hour. Figuring 6 calls per person : 77 / 6 ~ 12.83~.

Lets round it up to 13 people. You would need 13 people each hour to handle the extra call flow. 13 people * $12 = $156

$156 * 24 = $3744 : The payroll now needs to account for an additional $3744 per day in order to pay the overtime workers to handle calls.

$3744 * 5 days = $18720 : For a 5 day work week, handling the extra calls would cost the contractor $18720 to pay the employees.

If the Owning company pays up their amount, the contractor would have this : $18720- $12256 = $6464

$6464 dollars per week that has to come from somewhere.


******************

Number Crunching :: The Technician Approach

So, lets take the same numbers and give them tech support training. 192 people, but because of their skills they are going to get paid $10 an hour in order to take calls.

So, 192 people * 10 dollars = $1920 :: $1920 dollars * 40 hours = $76800 :

The payroll would run $76800 for each employee to work 40 hours a week. $76800 / 5 days = $15360.

The expenditure figuring a 5 day work week would be $15360. $15360 / 9216 = $1.66~ :: so each call is worth about $1.66 dollars

Now, lets screw the numbers all up. Lets say that 5% of the 9216 daily callers have to call back in. That is about 461 (460.8) extra calls coming back in.

If the company that is paying for the calls pays for the extra 461 calls, they'll pay $1.66 per call. That would be about $765 extra from the Company, per day. Multiply that times 5, and the additional 5 day work week cost is about $3826.

If the contractor taking the calls takes the hit out of their own pocket for the call backs, say, they pay time and half to handle the calls... lets run those numbers.

10 / 2 = 5 :: 10+5 = 15. So, $15 per hour. 461 calls / 24 hours ~ 20 (19.2) calls an hour. Figuring 6 calls per person : 20 / 6 ~ 3.3~.

Lets round it up to 4 people. You would need 4 people each hour to handle the extra call flow. 4 people * $15 = $60

$60 * 24 = $1440 : The payroll now needs to account for an additional $1440 per day in order to pay the overtime workers to handle calls.

$1440 * 5 days = $7200 : For a 5 day work week, handling the extra calls would cost the contractor $7200 to pay the employees.

If the Owning company pays up their amount, the contractor would have this : $7200 - $3826 = $3374


The Analysis

So, overall, using the "cost effective" employees, the Owning Company would pay $61440 + $12256 = $73696

Using Trained Employees and you get this : $76800 + $3826 = $80626

On paper, the cost effective employees do appear to be more cost effective, costing almost $7000 ($6930) less.

However, in the case of the Contractor, hiring the Cost Effective Employees means paying $18720 in order to handle overtime charges. In the case of higher Technicians, the overtime charges are only $7200, less than half.

Okay, so why 192 people. Why $8 an hour and $10 an hour?

The 192 is easiest to explain. I originally started running these numbers based on a $12 an hour paycheck, which is what I was earning when I left Sitel. 12*16=192, which was easier math to keep track of. I went with $8 and $10 because I started at $8, and those numbers should be about average for today. The final result shows just how close the costs can be.

The question that needs to be asked is this: Is it really worth it? Is it really worth saving $7000 to hire non-technicians to handle technical calls?

In the case of the trained technicians, you error on the side of "worst case" scenario. It is presumed that only 5 percent of the callers need to call back. Realistically speaking, 99% of the calls received in HSI or Computer support are 2 minute quick jobs, things like "Plug the Power Cord" in.

In the case of the "cost effective employees" you presume only a minor 20% call back ratio. That number, while pulled from SilverBears post, errors significantly under the call back ratio I expererienced. Take it from somebody who worked in tech support with the "Cost effective method". It was not uncommon when I took a call for there to be 6 or 7 entries already logged into the database for that day. Multiple times when I had the customer explain to me what was going on, I knew exactly whose login I would find in their account. It was not a joke for those of us technicians in Sitel that other call centers did not have a bloody clue what was being told to the customer. When I left Sitel the callback ratio from other Call Centers was somewhere around 40-45%. Almost 1 out of every 2 callers would have to call back in for assistance.

Okay, fine. I am just one tech. I didn't talk to everybody. I did not handle all the calls. I did not see everything. All I knew is that if I took 70 calls in one night, more than 30 would be from customers calling back in. I filed several emails to our training staff and management to please get a hold of the other call centers and get their acts straightened out.

I personally wrote Training Documentation and user manuals for Cox Communications Tier 1 HSI, and sent the data to our own Trainer, to San Diego, to Atlanta, to Tier2-HRD, to CTC, to Sykes, to Texas, to everywhere I could, because I was fed up with Script Kiddies screwing around with clients.

The Breaking Point : Further Calculations on Cost Effective Compounding

I reached my breaking point after 3 years.

At what point do the consumers break though. With the cost effective employees, 1843 people could be expected to call back in with problems. What happens if they don't get their problem resolved on their second call back? What happens if the customer has to call back 3rd, or 4th time? Lets keep the 20% ratio.

20% of 1843 people call back in for a 3rd call. That is about 369 (368.6) consumers calling back.

20% of 369 call back for a 4th time. Now we have about 74 more calls (73.8).

9216 calls + 1843 calls + 369 calls + 74 calls = 11502 calls.

At $1.33 per call, the owning company is looking at a charge of $15296.66

The contractor now has to account for the additional overtime to handle these call backs.

369 calls / 24 hours ~15 calls an hour (15.375). 15 calls an hour / 6 calls per person ~ 2.5 people

So, the client now needs an additional 3 people each hour, to handle the 3rd call backs.

That's a total of 13 + 3 = 16 people per hour to handle the call backs.

73 calls / 24 hours ~ 3 calls an hour (3.08). 3 calls an hour / 6 calls per person ~ .5

So, finally on the 4th call back, again presuming 20% ratio, do we reach a point where The number of people already accounted for will be present to handle the calls.

So, 16 people at $12 an hour = $192 per hour. $192 * 24 = $4608 per day.

$4608 * 5 = $23040

$23040 is the total amount of money that needs to be set aside for the "cost effective" call center to pay the overtime for staff to stay and take the calls.


******************

The Breaking Point : Further Calculations on Technician Compounding

Now lets run the same call backs with our Technicians.

5% of 461 people call back in for a 3rd call. That is about 23 (23.05) consumers calling back.

5% of 23 call back for a 4th time. Now we have about 1 more call (1.15).

9216 calls + 461 calls + 23 calls + 1 call = 9701 calls.

At $1.66 per call, the owning company is looking at a charge of $16103.66

The contractor now has to account for the additional overtime to handle these call backs.

23 calls / 24 hours ~1 calls an hour (0.96). 1 call an hour / 6 calls per person ~ .2 (0.16).

On the First go around we had a needed number of about ~3.3 people to handle the calls, and it was rounded up to 4. With that round up, we've already accounted for 5% of the call backs on 5% of the original calls.

The cost to the contractor doesn't change. Paying $10 per hour to trained techs with time and half over time, and accounting for a 5% callback repetition, the contractor is covered with just 4 techs working an hour of overtime each hour.

****

The Final Crunch

The Cost Effective Business model winds up with a charge of $61440 + $23040 = $84480

The Technician Model is still at $80626

Suddenly, the Cost Effective Model... isn't. It is MORE EXPENSIVE.

Now, I appearently am dead wrong about this, as the Accountants running Tech Call centers don't consider it worth the cost to pay out for Technical Staff.

In my example, would it not be better just to pay that about $7000 out and get technicians that get things done on the first try, and not risk having to hire extra staff or pay overtime? According to an Accountant, I would be wrong.

Oh wait... I wasn't wrong. Cox Communications Sold their contract because Sitel went the cost effective route, and it failed.

The problem is, Accountants do not thing about things like "One Call Resolution." It is my opinion that such concepts are beyond their grasp. But even when playing the numbers game, the Cost Effective method just isn't viable, period.

The Public Explanation

LnoyBoy brought up the public explanation given for going the cost effective route:

Really interesting post.

I guess I'd been fed a line of bull. I was told that the script readers were in place
to take up the "are your cables loose?", and "is it plugged in?" type questions, so
the real techs wouldn't have to deal with them.

That... really isn't a line of bull. Trust me, if you ever get bored enough to read my old Live Journal ( saist.livejournal.com ), I had to deal with thousands of callers each year that pulled a cable loose, kicked the power cord out, set the modem in standby, or couldn't identify the power cord.

Then there are/were the people who call in during an outage, while a "Your Area Is In an Outage" message plays over the phone system, and ask "Is there an outage?"

The real problem that Sitel ran into is that the script readers didn't know when to stop reading the script and start diagnosing the problem.

E.g. : Somebody calls in with a slow or Intermittent connection. First thing you have the customer do is run a couple of ping tests. Get a local server, get a server some distance away (like Google or Yahoo), then grab a server across the ocean (I went for theregister.co.uk). You look for the overall ping response and the variance in ping speeds. You check the modem's levels, make sure that the Signal to Noise ratio is within range. If anything at all looks wrong with the signals, you rolled a truck and got a line tech out there.

The problem was for us is that a lot of script readers were not even opening up the modem to check on the levels. Several would simply feed the customer some line of manure about their computer, advise taking the computer to a repair tech, and end the call. Others would just simple send the Caller right up to Level 2 support. Tier 2 takes one look at the account, sees no troubleshooting, and sends the customer right back down to Tier 1 (front line calls) for Troubleshooting. So, if you have a 10 minute hold time before the customer gets to Tier 1, they spend 10 minutes on the phone calling, 2 minutes talking to the front line rep, however many minutes waiting on Tier 2, then right back into the 10 minute wait time to talk with a Front line tech again. That's... 22 minutes at a minimum, and the customer's problem hasn't been addressed at all.

However, as far as I can tell, accountants are either unable, or unwilling, to see these trends, or account for them. Accountants look at the "bottom" line of how much "has to be paid out" ... not how much "has to be earned."

It is my opinion that one of the reasons why Cox Communications sold many markets to Suddenlink communications at rock bottom prices is because of the Tech Support and Customer Service that the customers were getting. I still remember taking Las Vegas back in 2003 and having upwards of 45 minutes hold times. Northern Virginia? You have a Senator wait 30 minutes, and trust me, your ears will /bleed/.

Customers don't want that, and at some point, they'll draw a line. We worked our tails off to get the call times down and to get things fixed on the first try, because those of us on the front line knew that none of these callers had to call. None of them had to buy our product, or our services. It was entirely voluntarily up to the customer, and if their product was not being delivered, they would take their money elsewhere.

If you are a company, in general terms customers do not have to buy from you. When the customers stop paying for the service, what happens? Was it worth possibly saving under $10,000 on a service contract when customers might cause an overall loss of $75,000,000 (not singling out any now owned Suddenlink market here).

From a Service standpoint, a balance needs to be struck. The techs need to be polite to the callers, but techs still need to be techs. A lot of Accountants running Call Centers just don't get that.

No comments: