General Dynamics - Pathfinder Ad

9-1-1 Magazine: Managing Emergency Communications

Priority Dispatch

Navigator 2018




Stratus Technologies

CAD, NG911 & Records Management


Recording Systems


Facilities Planning and Design

First Contact 911

Training Trends & Tactics




Holland Co     
Mobile Command Vehicles



The Numbers Game: Are Score-Based QA Systems Truly Representative of Dispatcher Performance?

Author: A 9-1-1 Magazine Special Report

Share |

A 9-1-1 Magazine Special Report

Originally published in our April 2008 issue.

In a profession as critical – and criticized – as our contemporary 9-1-1 occupation, adherence to policies and procedures becomes extremely important. This is especially true for agencies that provide some measure of post-dispatch and/or pre-arrival instructions to callers, directing the caller to actively do something to stabilize a patient or begin to mitigate an emergency before the arrival of the dispatched first responders. Evaluating performance and compliance with protocols and procedures becomes as necessary as the provision of the protocol itself, since doing so incorrectly could wind up harming the patient or exposing the agency to liability.


Thus were born quality assurance (QA) systems and their close cousins, quality improvement (QI), whereby performance measures are targeted and evaluated for compliance, and whereby non-compliant performance can be revealed and corrected. The challenge in QA is to come up with a system performance evaluation that is not perceived as punitive. Perception is often based on how the individual feels, and constant reviews of performance may not always feel good, thus the QA process is often perceived as punitive, unnecessarily nit-picky, and based on an armchair-review, performed without the specific circumstances and outside influences under which the call was first taken by the dispatcher.

There are a variety of quality assurance/improvement systems in place in 9-1-1 centers, although their use is a relatively new enhancement to 9-1-1 center management. Those agencies that use specific protocol or guidecard systems, such as the Medical, Fire, or Police Priority Dispatch Systems from Priority Dispatch Corp (developed and maintained by the National Academies of Emergency Dispatch), the Multi-Purpose Emergency Dispatch System from the APCO Institute, or Powerphone’s Computer Aided Call Handling (CACH™), use QI programs that are developed to work in conjunction with those specific systems. Other centers may have developed or adopted other forms of QI programs in house. Whatever the product in use, what QI programs have at their core is a methodology of call review and evaluation measured against specific and understood performance targets.

In order to generate a comprehensive and balanced report on the use of score-based QI/QA systems that are used by 9-1-1 dispatch centers to evaluate protocol compliance and/or dispatch procedures and services, 9-1-1 Magazine queried dispatchers, supervisors, QI/QA personnel, managers, and EMD system providers. Our goal was to examine the value of score-based evaluation to agency management as well as the affect such systems may have upon dispatch staff and operations. What follows is a compilation of the perspectives of our respondents.

Forms of QA

“The purpose of a QA program is to ensure that dispatch guidelines are being followed and department policies and procedures are being adhered to,” said Lesli Prado, supervising public safety dispatcher, Livermore Police Department (CA), which has a newly-developed in-house QA program. “A sampling of calls will be reviewed each month and evaluated for compliance to our Communications and Policy Manual.”

Minneapolis 3-1-1, the Minnesota city’s non-emergency PSAP, developed its own QA process as well. The agency uses a software program called Higher Ground to record all voice and screen capture images from incoming and outgoing calls at the center. “Higher Ground is set to randomly push a set number of phone calls to each of the four people on the quality assurance team,” said Marc Dronen, knowledge base and quality assurance manager. The agency developed a score sheet in-house and uses that to score each of these calls. “Scores are recorded and I look for scores that fall outside of tolerance. The QA staff meets once a month to discuss the calls – and specifically the ones which are out of tolerance.”

Supervisors at the Palm Beach Gardens (FL) Police Department review calls on a weekly basis, evaluating call handling against the city’s complaint taker’s guide. Numerical scores are not used, but dispatchers receive a written report of the evaluation. “High priority calls are checked and gone over with call taker and/or radio person on how call could have been handled better or if call was handled in proper way,” said dispatcher Mariehelena Morris.

Fairfax County (VA) dispatchers use the APCO EMD system and an associated QA program from recording system provider Mercom. “Review forms are designed to be rated by a yes/no to each area evaluated,” said John Korman, Communications Supervisor. “I do not want the review forms to imply or be perceived by the dispatcher as an evaluation.”

“Due to the customization of the guidecards, APCO doesn’t have an ‘official’ [QA] program,” said APCO’s Courtney McCarron. “All agencies are supplied with guidelines and instructions to develop their own program. In addition, all EMD instructors are supplied with generic, customizable forms to develop their own call review formats.” APCO’s Multi-purpose Emergency Dispatch System (MEDS{™}) software has a QA component included with the software purchase.

The Aurora (CO) Public Safety Communications Department manages its homegrown EMD program with an equally self-developed QA system. “The state of Colorado has very broad, shallow requirements regarding EMD,” said Dave Dooley, dispatch supervisor. “We wrote something more in-depth.” Their QA program is built around the relative smoothness with which their EMD program is designed, scoring dispatchers with a “1” if they are incompliance with the EMD protocols, or a “0” if they are not. “As long as the call taker performs in a reasonable manner, with due diligence, they are in compliance,” said Dooley.

Other protocol systems are far more stringent in their adherence to specific scripted verbiage and dispatchers lose points in review scores for not following the text verbatim. Silicon Valley PSAPs like San Jose and Sunnyvale (CA) are two of many PSAPs using the Medical Priorities Dispatch System, which is supported by a meticulous QA process to ensure that medical instructions provided to 9-1-1 callers are given exactly as scripted. A score-based review and analysis system was devised to evaluate protocol compliance, and the NAED has developed a training program for Dispatch QA supervisors (ED-Qs) to run the QA unit (see sidebar).

Dispatch Scripts and Scores

The basis of score-generated dispatch QA can be found in the prevalence of EMD, pre-arrival instructions, and scripted protocols, particularly those of medical priorities. The National Academies of Emergency Dispatch, which qualifies and manages the Medical Priority Dispatch System, asserts that a scripted protocol is essential to emergency call processing because of the non-visual nature of the process. “With specific regard to medical call processing,” explained Brett Patterson, a senior EMD instructor and an academics and standards associate for the NAED, “the EMD is unable to take advantage of the same patient care assessment techniques that the field provider is trained to utilize, i.e., sight, smell, and touch, which trigger various assessment and treatment responses when the provider is in the physical presence of the patient(s). Because these scene and patient assessment tools are absent in the non-visual realm of EMD, the call taker must rely on what is provided by the untrained caller, who must be specifically prompted to provide the most appropriate information in a limited amount of time. Without the benefit of physical scene assessment tools, the EMD must be prompted by scripted protocol so that essential elements of the situation are not missed.”

Patterson explains that this is the reason for the NAED’s strict adherence to scripted interrogation and instruction that must be read verbatim in their EMS, fire, and police protocol systems. “Specific wording of the protocol becomes absolutely critical in the non-visual realm of emergency dispatch,” he said. “Absolute caller understanding is essential to obtaining accurate information and relaying precise instruction without the benefit of visual confirmation of understanding. For example, the scripted instruction ‘Let him rest in the most comfortable position’ has a very different meaning than the innocently paraphrased instruction ‘Get him in the most comfortable position,’ as the latter becomes invasive and may actually be detrimental to patient care. Examples of this fact are virtually endless making exact wording of non-visual interrogation and instruction critical.”

Scored QA, then, becomes equally critical in ensuring compliance to the scripted protocol. Questions or instructions that are not read verbatim, or those that are missed or are freelanced, result in low scores. “Emergency dispatchers who, through adequate initial training and continuing education, understand the science of remote interrogation and instruction, not only accept the “letter-of-the-law” approach of emergency dispatch QA, they welcome it as it provides the essential feedback necessary to improve their performance, which they naturally want to do,” said Patterson.

Strictly Speaking

By their nature, QA programs are designed to be strictly implemented, especially those validating compliance to specific post-dispatch instructions meant to be relayed to callers. Like paramedics who must comply with specific medical protocols and procedures, dispatchers providing pre-arrival medical instructions to callers must be evaluated for adherence to those protocols.

The QA system used by Minneapolis 3-1-1 combines yes/no questions with a rating of between 5 (Excellent) and 0 (Fail). “We add the points up and see what percentage of points were achieved,” said Marc Dronen. “Each of the four types of calls has a different range to have the call classified as excellent, good, fair, poor, or fail. The score-based system provides an across-the-board scale of performance. This also takes out much of the subjective nature of rating calls.”

“Rather than the ‘I like you so I’ll give you high marks’ or ‘I don’t like you so you’ll get mediocre marks’ subjectivity that prevails in many communication centers, our evaluation process has become performance-based by measuring compliance to our standards,” said Michael Spath, an NAED ED-Q instructor and administrative senior public safety dispatcher for the Sunnyvale (CA) Deptartment of Public Safety. “The scores are not the evaluation; they are the result of an evaluation. The evaluation is based upon compliance to standards. It’s a fine point, to be sure, but it is vital to understand the difference. The scores are used to establish a baseline from which we all can correct and improve behavior, if needed.”

“The system we use has defined specific standards that must be followed in order to ensure proper patient care, then assigns a numerical value to measuring that care,” said Doreen Hargrave, supervising public safety dispatcher (and former ED-Q), San Jose Fire Department Communications. “The reviewer has to follow the prescribed rules when checking compliance in order to assure standardized delivery of care. Without standardization, which is perceived as strict, the measurement of performance is invalid. More importantly, without standardization, the application of the protocol will not be consistent, and customer service (patient care, safety instructions) will be different based on who answers the call.”

“The call review system should evaluate the telecommunications skills of the EMD as well as the use of the guidecards,” said Courtney McCarron of APCO. “Performance trends at the agency level can be addressed with continuing education and progress through the steps with policies and procedures.”

The Spirit, The Letter, and the Review

By its nature, a QA program is designed to address specifics of the protocol or guidecard system, or the call-taking procedures of the agency. However, some dispatchers feel a little more discretion in call review – a kind of “spirit of the law” versus “letter of the law” interpretation of performance evaluation, at least where the system would allow it – would be have big benefits towards dispatcher morale and well being. While many of these numbers-based QA systems generate rich data for management and administration, what many fail to recognize is the effect such stringent scrutiny has upon the dispatcher doing the job. Some agencies suffer from low morale, their dispatchers feeling beaten upon each month when minutiae-laden call reviews tell them where they have failed.

“Following the letter of the law as opposed to the spirit is what degrades the value of our evaluations,” said Beth Schultz, a communicator with Dane County (WI) Public Safety Communications, a user of NAED protocols. “Being told I should have asked a question to which the caller couldn’t possibly have known the answer does not assist me in doing better next time.” Nor does being aware of what is obvious, she added. “We are told that ‘anything not asked is not discovered.’ So if we get a call about an alarm call at, say, Shopko, where we shop every week, we still have to ask, ‘How many floors or stories is the building?’ even though we know darn well it’s a single-story market. If we don’t ask, we are downgraded. ‘Obvious’ is not an excuse.”

“The way things are with the letter of the law is that we are the tools and the cards are using us,” said Patty Hubbard, a 23-year veteran dispatcher currently with San Jose Fire. “Sometimes, one has to put some human emotional input into the call in order to get the information needed. Sometimes, following the letter of the law does not take that into account.” Hubbard relayed one call in which she relied on her instinct to ask an additional question not called for in the protocol for an alert young woman reporting lower abdominal pain (“are you pregnant?”). Receiving an affirmative answer resulted in an entirely different set of questions and an emergent response to the call whereas the simple abdominal pain complaint would not have. A strict review of protocol compliance without some understanding of why she felt compelled to ask the caller the “freelance” question would have scored her down for asking a freelance question, when in this case it was, in fact, appropriate. “If we as dispatchers are not able to put our years of experience into what we are doing, we may as well be doing something else,” Hubbard said. “The human factor in a call is very critical and the letter of the law does not allow for that.”

“Sometimes the evaluation is too concerned with minutiae instead of concerning itself with the fact that all the basic and pertinent information was gathered,” said one Palm Beach Gardens dispatcher.

“The major benefit of QA is having a clear indicator of the service that is being provided to our customers,” said Hargrave. “I have a specific measurement of the EMD’s ability to follow the protocol which best delivers care to callers/patients. I can clearly point out EFD’s performance to senior staff in an effort to dispatch our rolling stock in the most effective manner. It also provides a tool to track trends in the use of the protocols for training as well as making changes to the protocol.”

But it’s not all about cold numbers, according to several dispatchers facing such scrutiny unhappily. The goal in any QA system is to provide feedback in a positive manner, but this can be challenging when conveying negative results or negative scores. Firm score-based QA systems inevitably result in perception gaps between the evaluators and the evaluated. The latter tends to feel overly-scrutinized while the former embraces the opportunity to map out trends and gather data to support overall performance or justify accreditation status.

“It’s important to consider how much a person’s ability to accept feedback plays into this process as well,” added Spath. “Some are better than others at accepting objective criticism.”

Quality Assurance and Discipline

The challenge in QA is to come up with a performance evaluation system that is not perceived as punitive. Perception is often based on how the individual feels, and constant reviews of performance may not always feel good, thus the QA process is often perceived as punitive, unnecessarily nit-picky, and based on an armchair review performed without considering the specific circumstances and outside influences under which the call was first taken by the dispatcher.

“My opinion is that it is malicious to score someone down on a call because they asked how old the patient is before asking if you are with the patient,” said dispatcher Brenda Jaime, San Jose Fire. “Emergency calls are not math homework. How I interpret the information/situation is not the same as a person listening to a recording of me three weeks later sitting in their office, wearing headphones that block out all noise and distractions.”

The purpose of QA is supposed to be proactive and educational, although it is not always perceived as such or delivered as such. The ongoing review process can feel like harsh micromanagement to the console jockey. Delivered in a positive and an understanding manner, with a goal toward guiding dispatchers toward excellence, too many well-intentioned QA programs can be sabotaged by a mechanistic, heavy-handed criticism of minutiae on the part of the Qs, and self-conscious misunderstanding on the part of dispatchers.

“We preach training and education,” said Aurora’s Dave Dooley. “We show the employee trends in their calls. By seeing consistent and repeated deficiencies, the employee and the reviewer can develop a program to assist the employee in erasing the deficiencies.”

“Simply put, call takers who think their QA process is overly critical or punitive are victims of a poor QA process,” said NAED’s Brett Patterson. “Unfortunately, many QA programs are run by poorly educated auditors, supervisors, and managers who have not had the benefit of an adequate QA education. While measurement of performance is essential to the QA process, it is only a small part of the QA process. Perhaps because auditing and scoring is the easiest of the QA tools to master, it is often seen as the only process necessary in a QA program. In reality, how the data is analyzed, fed back to employees, and used to train are actually just as essential and, without these additional skills, the measurement process becomes futile as it is most often used only to tell employees what they are doing wrong, in many cases punitively.”

“I feel QA is a good tool for learning because you may not realize you are making a mistake,” said dispatcher Mariehelena Morris, Palm Beach Gardens PD. “When you are told that you handled your call very well that also helps moral.” Another Palm Beach Gardens dispatcher held an opposite opinion, feeling the system is too negative. “Very seldom is a comment put in a QA that is positive. The QA we follow is either a ‘Yes you followed exactly’ or ‘No you didn’t follow exactly.’ There are calls that come in where it is not possible to get the questions asked exactly as scripted.”

There may ultimately come a time when QA results in discipline, if the dispatcher has chosen not to comply, or the supervisor has decided not to hold the employee accountable, those choices must be addressed by the progressive discipline process. While the QI process may have identified this particular issue, the discipline process must be used to correct the behavior. At that point, it is no longer an issue for the QI process.

“Regardless of how a deficiency is noted (scored review, supervisor observation, customer complaint) it can’t be ignored,” said Hargrave. “There are many areas of the job that are not scored as a measure of the performance. Problems in those areas also need to be identified, training provided, and if there is no response to training, it is incumbent upon the supervisor and the agency to ensure appropriate service is rendered to the customer.”

On Being Graded

Even the most well-intentioned QA program cannot escape the simple fact that it results in a system wherein dispatchers are constantly being graded and presented on a weekly or a monthly basis with the Comm Center equivalent of either a gold star or a “frowny face” sticker. While feedback is generally beneficial, dispatchers report that the constant critiquing of performance can result in low self-esteem and a consequential lack of confidence in the system.

“Being a seasoned dispatcher and having your every move monitored is so degrading,” said Patty Hubbard, San Jose Fire. “Even if you scored high, there always seems to be a ‘great job, but...’ added to the review. Now when you come into work and you see that you are assigned as a call taker, your stress level shoots sky high. You are no longer taking calls and putting the customer first. You are taking calls and thinking about the score first.”

Beth Schultz agrees: “When we hang up on a difficult call, again having done what we believe to be the best job we could under the circumstances at the time, our first thought isn’t for the patient or the situation. Our first thoughts are ‘I wonder what kind of score I’m going to get.’ Not a very good attitude for reviews that are supposed to help us improve!”

This is an area where some dispatchers feel their reviews tend to become overly critical, even when QA supervisors go out of their way to emphasize what was done correctly on a call that was ultimately graded as out of compliance. The “good job, but…” syndrome is problematic, since whatever criticism follows that “but…” invariably invalidates whatever positive comments were emphasized before it. Dispatchers quickly see through these kinds of reviews and find little value in them. QA personnel need to understand how their method of criticism and call analysis is presented – and perceived – by the dispatchers they are evaluating. As much empathy as call takers have for their callers should be exhibited by the QA staff reviewing their calls, without discounting procedure errors.

“Not every single call is always a yes or no type of call,” added Janna Faulhaber, Palm Gardens PD. “This system feels like more of a punishment and not a learning tool for making a better dispatcher. There’s not enough positive feedback.”

QA should look for patterns of call taking and not discount or discredit a call for one minor issue, agreed another Palm Beach Gardens dispatcher. “Some leeway has to be applied in the evaluation process in term of individuality. Not all situations can be addressed by the same questions asked in the same order or in the same way. The evaluator needs to realize that their way of handling a call may not be the only way.”

“The score is simply the essential measurement component in a much larger process,” said Brett Patterson, NAED. “It is very clear, as is evidenced in highly compliant communication centers, that good customer service actually increases protocol compliance as callers respond so much better to caring call takers. Customer service is an essential component of the QI process.”

Building a Better QA System

What would be the ideal kind of QI system for a 9-1-1 Center? Most managers and dispatchers agree that a proactive QA must be presented and perceived as a positive thing. This includes educating dispatchers about the philosophy behind the protocol or guidecard process so that they better understand why the system is designed the way it is, and in allowing leeway when appropriate. Using the QA process to discover ways to improve the protocol or guidecard system is also very much a part of quality improvement.

“It is imperative that feedback is not all negative,” said APCO’s McCarron. “Positive feedback plays a large role in the success of the system. Individual reviews should not be made public to the staff unless they are positive.”

“We recognize that our point structure for our ratings should be greater,” said Marc Dronen, Minneapolis 3-1-1. “Instead of providing a one point spread between excellent, good, fair, poor, the difference should show clearer how the calls are falling into categories.”

“To make these systems most beneficial to dispatchers, they need to allow discretion in reviewing by looking at the big picture,” said Beth Schultz. “Did the ambulance get sent to the right place at the right speed for the right chief complaint? Were the post-dispatch/pre-arrival instructions given appropriately in a way the caller could understand? That’s the big picture. Not: Did the call taker word a question incorrectly? Did a call taker skip a question because it was clear the answer would be ‘unknown?’ Did the call taker skip a question that was clearly obvious? Did the call taker forget to say ‘Put away your family pets?’ Evaluations that include this minutiae do not help us become better call takers.

“By far the most important factor for auditors and managers of the QA process is education,” said NAED’s Brett Patterson. “It is essential that the agency is committed, from the top down, to QA/QI education. This commitment extends well beyond teaching auditors how to score a call and provide a report card and a scolding to employees. Once an agency is committed to implementing an excellent QA program, all stakeholders of the process, including line employees, need to be educated in the various components of the process.

“This begins with QA philosophy and includes not only scoring techniques but the mastery of data analysis tools, feedback techniques, and understanding how to interpret data and use it to construct effective continuing education.”

“Supervisors and managers need to balance the feedback and make sure the negative doesn’t overpower the positive,” said APCO’s Courtney McCarron. “Dispatchers need to remember the training in liability and risk management suggestions they received in their EMD course. The call review and QA process must become an educational opportunity for everyone.”

QA for specific programs, like EMD or call taking in general, can be a part of an overall performance evaluation, but only a part. Agencies that use call taking QA results as a measure of their employees’ overall performance evaluation are most likely ending up with misleading evaluations. “It becomes too easy to rely solely on the QI scores and neglect appropriate evaluation of other performance,” said Deanna Mateo-Mih, one of San Jose Fire’s long term ED-Qs. “Agencies are not evaluating overall employee performance using objective, performance based, quality management methodologies. This means that the only quantitative score available is from the protocol systems. The result is that there is an over emphasis on protocol scores and not enough on other areas of performance.”

“In general, I think statistical based system is the most objective way to determine protocol compliance, but there is a lot more to the job than just following the protocols,” said Battalion Chief Jeff Clet, San Jose Fire, who helped start the department’s QA program. “Managing radio traffic, interacting with the other dispatchers, handing stressful calls, telephone etiquette, customer service, etc. There are many other components to a dispatcher’s job and employee performance shouldn’t be evaluated simply on one component.”

“I have learned no one is perfect, and never will be,” said one dispatcher from Palm Beach Gardens. “Even if you know the guidelines, you are still going to make errors. I really feel that in this job, there is no exact science.”

“With regard to line dispatchers, educated facilitators will understand the importance of employee involvement, empowerment, and education, rather than punitive measures, and the dispatchers will respond appropriately,” concluded Patterson. “Employees come to work wanting to do a good job. Unfortunately, in many cases, the destructive practices of a poorly educated management tend to destroy this innate desire to perform well. A properly implemented QA program will foster performance improvement, and will do so by taking advantage of the natural desire to do a great job through empowerment and education.”

This article was part of a comprehensive overview of Score-Based Quality Assurance in Dispatch Operations.
see also:

The Origin of Score-Based Quality Assurance

The most pervasive method of QA call review is that wherein calltakers are given a numerical score rating based on their performance in complying with protocol or procedures in their call-taking performance. The NAED’s Jeff Clausen, who created the first EMD program in the early 1970s, described for 9-1-1 Magazine how they chose this method of call review:

“It became apparent in the mid-1980s that compliance to protocol was hit and miss and the only way to assess correctness of following protocol was to audio review calls,” Clausen said. “Initially, 5 to 10 cases were brought to a meeting and listened to. To reduce the inter-rater variations (often profound) a review template (checklist sheet) was developed. In order to quantify the results of more than one case, a scoring system evolved to allow for an accumulation of cases. The score-based system allowed for an apple-to-apple comparison of cases and EMD performances. The most interesting thing about such a system is that when we fed the scores back to the EMDs individually, and gave them something to compare them to (their shift, other shifts, other centers, the NAED accreditation minimums), their scores went up rapidly. Without weekly to at least monthly feedback, the scores dropped.

“Early on, the use of a committee to review cases was abandoned as it was cumbersome, often argumentative, and could not review enough cases to give significant amounts of feedback to the on-line EMDs. From this the EMD-Q concept evolved, with specially trained individuals reviewing large numbers of cases, coalescing the data, and providing feedback to both the EMDs and management. It should be noted that of the last 25 agencies that attained NAED accreditation, in all six areas scored (the seventh being the total score), nearly all scores were well above the 90 to 95% minimum, and the vast majority were above 97%. The average scores when QA review begins is usually a wide variation between 50 and 80%.

“The score-based QI process has been a spectacular success. Nothing else we have tried comes even close to improving compliance to protocol as quickly or to near as high a performance level as routine score-based feedback.”



Show: Newest | Oldest

Post a Comment

Log in or sign up to comment

9-1-1 Magazine is a Sponsor of the California Mobile Command Center Rally

Send mail to with questions or comments about this portal.

© 2010-2017 9-1-1 MAGAZINE and The content of this portal is the property of 9-1-1 MAGAZINE and  We encourage government public-safety agencies to share any content with their staff, however, all others must not duplicate or modify any content without prior written consent of 9-1-1 MAGAZINE. Email for permissions. For more information, read the Terms of Service. Continued access of this portal and system implies consent to the above statements and those maintained on the Terms of Service.

Powered by Solata

MCM Consulting Yellow Submarine Marketing

 Team Rennick

Holland Co