Browse Content by Topic:
A Different Kind of CAD System Security Update: Best Practices in Legacy Migration
Author: Michael D. Falkow, City of Inglewood, CA
Copyright: 9-1-1 Magazine, Feature Content
Call it ironic – but it wasn’t funny. It was 2009, and a simple operating system security update took down the 9-1-1 call system we spent the past 19 months trying to migrate. Had this been in production, it would have put our city’s 110,000 citizens at risk. Eight hours went by before we were able to roll back the update and get the code functioning again. Clearly this wasn’t going to cut it. There was no way we could put a mission-critical system with this type of vulnerability into production. It was back to the drawing board after 19 months and nearly $230,000 of sunk costs, not including significant staff time.
Inglewood, California is in Los Angeles County, with an urban population and a very active 9-1-1 emergency dispatch. In fact, we get more than 250 calls for service each day, greater than all of our neighboring cities combined. As the Assistant City Manager and CIO, I knew firsthand how 9-1-1 calls can save lives and mitigate further risk to citizens and law enforcement personnel. When I joined the City’s information technology and communications department in 2003, the Computer Aided Dispatch (CAD) system became a priority for me.
The 9-1-1 system was deployed in the 1980s and had served us well, but the mainframe it ran on was costing us $120,000 a year in licensing and maintenance costs and another $210,000 in associated personnel costs. And, every year the possibility grew that there’d be an unanticipated system crash downing our 9-1-1 system.
In 2005, we began the decommissioning process, developing a strategic plan and issuing RFPs for commercial-off-the-shelf (COTS) products. Our first thought was that off-the-shelf was the way to go, replacing everything we had before with a new, sleek system that would integrate across our entire department. During research we realized the price tags were around $1.5 million, which was beyond our budgetary reach. So we initiated a plan that called for the conversion of the original CICS/COBOL code to a more modern .NET language running on a Microsoft® Windows® Server platform. This would get us off the mainframe and away from the looming possibility of failure. So began what was supposed to be a six-month project that lasted for almost two years culminating in the 2009 security update incident that dashed our hopes for a successful code migration.
We started by taking a second look at the requirements we had for CAD. For many of the city’s systems, the migration process was more or less straightforward – but the CAD system posed a greater challenge because we couldn’t afford any downtime. When a 9-1-1 call is received at one of our eight emergency control center desks, the CAD system transmits the details instantly to every one of our 40 police patrol cars. It demands 24/7 availability and 99.999% reliability – which meant that we couldn’t do a traditional migration as we couldn’t have the system down for any period of time. Even a minute offline could pose a significant risk to the safety of the city’s residents and our officers. The migration would be like having to change the engines on an airplane while still in flight. For that reason, we turned to Micro Focus®, a company that’s been an expert in these types of migrations for decades. Ten months and less than half of the cost of the first failed attempt, we successfully emulated the mainframe environment on a Windows-based platform and took the CAD system online.
From this experience, here are the five best practices for a CAD migration I walked away with:
1. Understand Your Own Requirements
Most organizations work to define their requirements for the purposes of the vendor – however, it’s just as important for your department to understand those requirements. Our first attempt was focused – understandably – on transitioning the system to something new. But it wasn’t like a traditional system transition because it had unique requirements. When looking at updating any public safety technology, ask key questions in defining your requirements – don’t just ask “what kind of replacement do we want?” Ask: “how is the transition itself different from a traditional migration?” Looking for what makes your system unique and different – as well as focusing on the process of migration and transition, rather than the attributes of your ideal end-product – will save you a great deal of time and money in the long run – and it could even save lives.
2. Don’t Be Fooled by Conventional Wisdom
The conventional wisdom said that COTS systems would be cheaper – and for good reason: off-the-shelf systems are usually designed for the average user, have been tested and proven in multiple environments, and have benefited from economies of scale and distribution. When you already have an in-place infrastructure, COTS systems can be a costly alternative, as we discovered when new systems would have been 10 to 15 times the price of our partnership with Micro Focus, not including annual maintenance costs, which would have certainly been much higher with a COTS system. Question all conventional wisdom in a project of this scale.
3. Look at Total Cost of Ownership
Our mainframe CAD system was originally designed, built, and implemented in the 1980s, but that didn’t mean it was free for us to operate in 2005. In fact, the cost of maintaining the mainframe was costing our city over $300,000 a year. It’s usually assumed that replacing something will cost you in the short run but save you in the long run – and yet by implementing our migration with Micro Focus, even subtracting the cost of the transition, we saved $225,000 in year one. And we’re going to save $300,000 in all subsequent years. Bottom line: maintenance can be more costly than a transition, so don’t just look at the stand-up costs; always look at the total cost of ownership.
4. Change Course if Necessary
We learned a lot from our first failed migration attempt – but the most important thing we walked away with was a solid conviction that in order to be successful, you have to be willing to change. It’s easy to throw more money at a problem, or to try to stick it through – but don’t be afraid to be wrong. The point is to recognize your misstep as fast as you can, so you can quickly change course. In the end, we spent twice as much on our failed first attempt as we did on our successful second one, but the important thing was that the implementation itself actually worked. Remaining focused on the mission rather than any stubbornness, pride, or self-preservation is actually one of the greatest challenges – but it’s often the only way to achieve success.
5. Build a Partnership
Embracing the value of the partnership with a vendor is critical. When the going got tough, it was the trust and leverage of our partnership that saw us through to success. Micro Focus embraced the same iterative attitude; if something didn’t work the first time, they were swift to try something new – they even sent one of their senior engineers to tackle our problems on the ground. If you don’t have that kind of relationship from the start, it’s often difficult to keep the project focused.
Today the CAD system is stable, without a single issue since the launch. Our city now has a robust and resilient CAD system that runs on a far more cost-effective platform. Most importantly, it continues to deliver the functionality that our city’s dedicated first-responders need to protect the lives of our citizens…something you might call a different kind of security update.
Michael D. Falkow is the Assistant City Manager and CIO for the City of Inglewood, California. For more information on Inglewood PD, see http://www.inglewoodpd.org/; for more information on Micro Focus, see http://www.microfocus.com/