Browse Content by Topic:
Case Study: Upgrades in Allegheny County for Continued Uptime Performance
Author: Gary J. Thomas, ENP
Copyright: 9-1-1 Magazine, Feature Content
In my position as assistant chief of a high-volume 9-1-1 public safety answering point (PSAP), I’ve seen just how important time can be. Seconds can literally mean the difference between life and death. The applications that keep our call center and centers around the world up and running are the definition of mission critical. From a heart attack to a house fire, dispatchers need the ability to quickly determine the extent of the situation and send the correct units to assist the victim and/or mitigate the situation. Our environment is very dynamic and it’s critically important that our computer-aided dispatch (CAD) software runs around the clock. In a word, an outage would be catastrophic.
In Allegheny County, Pennsylvania, where I work, public safety telecommunicators handled 1.3 million emergency and non-emergency calls in 2011. The Center dispatches the 91 police, 167 Fire and 46 EMS agencies that serve and protect 130 municipalities in a 731-square mile area, the largest city being Pittsburgh. Our primary center consists of 20 call takers, 39 dispatchers and four supervisory posts. Needless to say, making sure our systems stay up and running, 24/7 is and always has been a critical priority.
Leaving Legacy PSAP Technology Behind
As with many of our peers, Allegheny County’s IT infrastructure was UNIX-based. Our systems had been operating reliably for nine years. More current and cost-effective products, however, had entered the marketplace that could help us integrate our two independent dispatch systems with improved resiliency and system-wide efficiency. The Agency’s software provider had re-written its products for Microsoft Windows, knowing that this was where future business growth would be coming from. Likewise, new industry-standard Intel-based computers were now available that were as reliable and powerful than the venerable UNIX technology we had become accustom to. Given the age of the infrastructure, the county determined that it was time for a technology upgrade, so in 2008 we decided to put the multi-million dollar contract out to bid.
Plans called for moving computer-aided dispatch (CAD) and other critical applications off legacy systems onto a Microsoft Windows-based application suite running on an industry-standard server platform. At the conclusion of the two-year effort, we successfully merged the two independent systems into a fully integrated, distributed and redundant system with both a primary dispatch center and an Overflow/Disaster Recovery site to ensure operational continuity regardless of circumstances.
The Bid for Uptime
The bidding process was rigorous, as it should be for any project of this nature. Prospective bidders had to complete a 700-point response matrix. We subjected a number of suppliers and their products to exhaustive and thorough analyses, one of which was a 99.99% uptime requirement as a minimum standard for the total solution. The new industry-standard hardware and software products from the County’s current vendors ultimately prevailed in the RFP process, with the added feature of virtualization. The Windows-based CAD system would run in virtual machines on fully redundant ftServer hardware from Stratus Technologies. The software package included a more robust mobile solution, hosted Web query, and integrated police records modules.
Solution Built to Last
The upgrade required updating both our Dispatch Center and Overflow/Disaster Recovery Center, which are located just five miles apart and are now equipped almost identically. A dedicated connection keeps the two centers synchronized with near-real-time log shipping. Most importantly, in the event additional dispatch resources are required or the primary communication center becomes inoperable, the disaster-tolerant configuration allows the overflow communications center to operate as an extension of the primary center or as a fully independent call-taking and dispatch facility with accurate up-to-the-second information. Just three computer-aided dispatch technicians oversee day-to-day operations and maintenance of the entire CAD infrastructure at the Overflow site.
The Primary Center houses a total of 18 servers, including two Stratus ftServer systems. Staff positions in the 63-seat center include 20 call takers, 39 dispatch and four supervisory posts. One ftServer system serves as the VMware ESX server, which host the two primary CAD applications on virtual machine (VMs). These two main VM servers run the database interface, Web interface host and complete message switch and interface programs. This enables the transfer of information from the 9-1-1 center to more than 700 field mobile users. About 300 mobile data terminals are connected at any given time, but that number can increase dramatically during large-scale incidents that require additional police, fire and EMS responders. The system also makes post-dispatch updates available to several different records management systems, as well as to more than 400 Web users on demand. The second ftServer system at the primary site runs one Windows Server that hosts the Microsoft SQL CAD database for Call History and complete system configuration and run-time settings. The Center’s 16 other servers run a range of less-critical application.
While our CAD system can always run basic functionality such as “enter calls for service” and “dispatch calls for service,” our entire support operation like remote CAD, system interfaces, mobile communication, Web query and police records management system is completely dependent on the reliability of our fault tolerant servers.
The Overall Mission
The job of first responders is challenging enough without having system interruptions that add seconds to onsite arrival in an emergency or prevents the calls from happening in the first place. Preventing unplanned downtime was a non-negotiable criterion in Allegheny County’s RFP process, and a primary reason for installing Stratus ftServer systems for its most critical applications. Just like every PSAP in the world, we simply cannot accept outages that can severely undermine our effectiveness at carrying out our mission. I’ll end this in the same light I started, because it’s absolutely true: seconds are a precious commodity in the business of life safety, property protection, and crime prevention and investigation.
Gary J. Thomas, ENP Asst. Chief/9-1-1 Coordinator has directed PSAP operations for the Allegheny (PA) County Department of Emergency Services for four years. He was Director of Communications for Union County, North Carolina prior to that. He is an adjunct instructor for APCO and NENA. He has also served for 23 years as a volunteer firefighter. Gary has a B.S. in Aeronautical Science, and an A.S. in Public Safety Communications.