9-1-1 Operations Continue Despite Lightning Strike to Lapeer County's Lifeline Dispatch Center
by Jeffrey Satkowski, Systems Administrator at Lapeer County Central Dispatch
On a Thursday, during a typical summer day in 2010, a violent electrical storm system was moving through the Midwest plains toward Lapeer County. Located 60 miles north of Detroit, Michigan, Lapeer County is a rural county covering some 663 square miles with a population of around 90,000. It is one of five counties that form the peninsula projecting into Lake Huron known as the Thumb, which is a sub-region of Flint and Tri-Cities.
Electrical storms always cause an increase in emergency calls to 911 as trees are brought down, localized flooding occurs and buildings are hit. Nationwide these calls come in to dispatch centers where trained public safety specialists mobilize emergency services to protect lives and property. Lapeer County is no different. If dispatchers become disconnected from emergency systems it impairs their ability to make vital and timely life and death decisions: when a disaster or emergency strikes, officials need to react immediately.
Lapeer County (MI) Dispatcher Tammy Knapp.
Lapeer County Central Dispatch’s Systems Administrator, Jeff Satkowski, was at the gym working out when his dispatch center called to alert him about the storm. Within seconds the emergency dispatch center and their radio tower were struck by lightning – everything went black. But with uninterruptable power supplies protecting the servers and emergency generators providing power Lapeer County was well prepared. Immediately following the strike all of the vital emergency dispatch applications hosted on their IT system were online and the dispatchers were able to respond to emergency calls.
However, the following day Satkowski arrived at work with a suspicion that the full effect of the strike had not yet been felt. Despite proper grounding, powerful strikes can weaken systems causing them to fail later on. Sure enough, during the afternoon the server responsible for the retention of all dispatch records crashed. This server ran voice recording software and handled all historical communications records, including telephone and radio traffic to and from the 911 center. While these records are important, they are not mission-critical to the operations of the dispatchers. After several attempts to restore the server were unsuccessful, a spare server was delivered and the installation began.
During the process to rebuild the voice recording server and get it back on line, their primary Computer Aided Dispatch (CAD) server also failed. The CAD server, running VisionCAD™ provided by VisionAIR, is a vital link in the provision of emergency services to the public. Without it the dispatchers have to revert to pen and paper which slows the whole process down and can be error prone.
Fortunately Stakowski had prepared for such an eventuality. VisionAIR’s VisionCAD System and its SQL database were protected against downtime through use of the Neverfail Continuous Availability Suite which maintained a second copy of the CAD system on an alternative server. Because the software also monitors whether systems are working and available to end users, it detected the failure of the primary server and immediately switched CAD operations to the secondary copy. As a result dispatch operations continued uninterrupted and public safety was not compromised.
The secondary server kept VisionCAD running for two weeks while the team sourced new hardware and got the primary server up and running again. Once this happened, Lapeer County worked closely with VisionAIR to load software, resynchronize the data from the secondary server and then use Neverfail to switch live operations back to the primary system.
“Had we not installed Neverfail earlier in the year, our dispatchers would have had to manually write out and process each incoming emergency call with pen and paper. This process would have been time consuming and inefficient – severely hindering Lapeer County’s ability to act on emergency calls,” said Satkowski. “For two weeks, we ran our dispatch services on the backup server, but eventually we were able to get the primary server fully online. Had it not been for Neverfail effortlessly switching us back to the primary system our dispatchers would have had to revert to manual processes at that time.”
Lapeer County (MI) Dispatcher Rebecca Wilson
The events during and following the storm proved that Jeff and his team at Lapeer County had been right to implement a business continuity solution to provide uninterrupted access to their mission critical VisionCAD system.
Working together with VisionAIR, Lapeer selected Neverfail to ensure the high availability of critical systems to meet the 911 needs of their residents. Neverfail protects VisionAIR’s application in the event of electrical storms, natural disasters or other catastrophes causing system overloads, failure or outages.
“We installed Neverfail in the spring of 2010 to protect access to the mission critical information stored on our CAD server. Shortly after the installation, Neverfail was put to the test during the summer storm – immediately paying for itself,” said Satkowski. “Prior to Neverfail we did not have anything installed to protect our dispatch services. During the storm, Neverfail kept our dispatchers connected to the vital information that they needed to respond quickly and effectively to 911 emergency calls. We thank God that we were able to successfully complete the project to protect our CAD server using Neverfail.”
Since installing Neverfail, Satkowski conducts routine tests every month when he is updating the Microsoft Windows servers to make sure the primary server fails over and back seamlessly.
Drawing on extensive industry experience, VisionAIR has developed an open, integrated software solution—the Public Safety Suite®. This proven solution includes Computer Aided Dispatch and Mapping, Records Management, Mobile Data, and Web-Based Data Sharing for 911 centers, law enforcement, fire service, and corrections. As a Microsoft® Gold Certified Partner and IJIS Institute Member, VisionAIR offers solutions built on industry standard platforms. For more information, see: www.VisionAIR.com
Neverfail is a leading global software company providing affordable data protection, high availability and disaster recovery solutions focused on keeping users productive. Neverfail’s software solutions enable users to remain continuously connected to the live software application irrespective of hardware, software, operating system or network failures. Neverfail’s mission of eliminating application downtime for the end user delivers the assurance of business continuity, removes the commercial and IT management costs associated with system downtime and enables the more productive use of IT resources. More information can be found at www.neverfailgroup.com