Browse Content by Topic:
Public Safety Technology - Aging Systems And New Acquisitions: Important Ideas To Consider
Author: Ronald H. Jayne
Copyright: 9-1-1 Magazine, Feature Content
When public safety software (computer-aided-dispatch [CAD], records management [RMS], mobile data systems [MDS] and automated field reporting [AFR] is installed, agencies rarely consider something called “the product lifecycle.” This lifecycle begins on the date the product is introduced into the marketplace; it “ends” when the product can no longer be effectively supported, is no longer competitive and in some cases, actually taken off the market. There are exceptions to the rule, but the average lifecycle for a typical public safety software product is estimated to be about ten years. If an agency acquires the system in year six of the product lifecycle, the agency can expect about four years of good use under normal circumstances.
As the lifecycle evolves, the agency will struggle to keep the system operational amid ever increasing annual maintenance costs. An aging system also suffers from “support deterioration” because older systems become more difficult and costly to maintain over time. To extend the lifecycle of a product, even though defects become more frequent and severe as the product ages, vendors will add functional capabilities called “life cycle extenders.” Life cycle extenders are new software components added to the system in hopes of keeping existing customers from moving to competitive products. Life cycle extenders are also used to retain marketability in the product even though they know the lifecycle is nearing its end. Examples of life cycle extenders include adding a windows front end to bring a windows “look and feel” to the product or adding functionality called a “dashboard” to enhance data retrieval capabilities for non-technical end users.
When a system is first installed, the annual maintenance costs seem “reasonable.” For the first few years, the agency will see “enhancements” to the product in each major release. As time goes on, however, enhancements usually become fewer and defects more common. The attention of the vendor’s technical staff is often focused more on defect repair than system enhancement. If the vendor isn’t doing it already, the agency begins receiving notice that a fee will be charged to install the next major release. If the contract does not protect the agency from these additional fees, the agency will soon realize that the installation fees for updates can easily exceed the cost of the annual maintenance fee. Some vendors charge installation fees beginning with the very first upgrade; this hidden cost is rarely disclosed to the agency until after system acceptance occurs.
Some vendors do not allow the agency to install updates on their own – installation by the vendor and the related fees suddenly are often mandatory for each major version release. If the agency cannot afford to pay for the update, they most likely will continue to receive “bug fixes” in return for the annual maintenance fee but future enhancements (major releases) are withheld unless and until the agency budgets for this previously unknown expense.
Aging systems also suffer from something called the deterioration of interface connectivity. To understand this phenomenon, we must first explain what an interface is and how it works in a public safety computer system.
An interface is a piece of software that automatically moves data from one system to another (i.e., a local RMS system updating stolen vehicle information in the state file). An interface can also be used to automatically query an external system (a local RMS user also checking the county or state database for warrant information). In performing these activities, the user does not need to log onto or access the other system(s) – it is automatic because it all happens in the background.
Let’s use the example of an E9-1-1 interface. Without an E9-1-1 interface, the dispatcher would have to look at a small 9-1-1 screen provided by the phone company, read the name, address, and phone number and then retype that information into the CAD call screen. With a 9-1-1 interface, this is all done automatically for the user; no user intervention is required.
Interfaces are designed to eliminate redundant data entry and maximize staff productivity. Public safety software vendors typically include the interface to E9-1-1 as a “standard” interface in the systems they sell. The support services provided by the vendor to maintain the connectivity and functionality of the interface to E9-1-1 are typically included in the annual maintenance fee – no additional costs are involved.
The interface to the County warrants system or to DMV, for example, however, is an example of a “custom interface.” In most cases, the cost to develop and install this interface is included in the price of the system purchased. What many vendors will not tell you is that the maintenance fee associated with maintaining the warrants and DMV interface is only available at an additional cost or worse, provided only on a time and materials (T&M) basis. This information is unknown until after the system is installed and accepted – it appears most often on the bill you receive for the first annual maintenance period that begins after final system acceptance.
The impact of this practice should be obvious. Interfaces that are considered “standard” by the vendor will continue to work and be supported so long as the agency pays the normal annual maintenance fee. Support costs for custom interfaces, however, are typically NOT included in the annual maintenance fee and as such, the vendor is free to charge whatever fees they decide. And since these fees are not typically disclosed at the time of the sale, the costs are not known until after the agency has signed off and accepted the system. If your agency was unfortunate enough to sign a contract that included an “automatic acceptance provision” you have no choice but to pay the additional fees or go without the upgrades.
A typical public safety computer system will have as few as three and as many as twenty-five interfaces. Only about five interfaces are considered “standard” by most vendors. All other interfaces are considered “custom” and subject to the costs mentioned above.
Here’s the bad news: Since a simple upgrade can “break” a custom interface so it no longer works, multiple custom interfaces become a moneymaker for the vendor and a frustrating, ongoing hidden cost for the agency. For a few years, the agency will agree to keep custom interfaces operational by budgeting additional funds to cover necessary costs. Over time, however, the costs escalate exponentially as it becomes more difficult for the vendor to keep the interface working and the agency will eventually decide the cost is not worth the benefit. To make matters worse, if the vendor decides they no longer want to support a custom interface, they simply drop it from their support program. Since most maintenance contracts are annual, the vendor is free to adjust their support program at will; the agency has no option but to either agree to pay the fees assessed or reduce the level of support they previously received.
Here is the ultimate impact: As costs escalate and/or the vendor discontinues support, the system deteriorates and end users begin to complain. The end of the product lifecycle has already begun.
Once the lifecycle is depleted, whether the agency realizes it or not, the system is earmarked for replacement. And when that happens, if your contract does not protect you, the vendor levies new software license and installation fees if you wish to acquire the replacement product(s). And consider this postscript: many vendors, sadly, continue to sell legacy systems even when they know the life cycle is depleted; not good news for the unsuspecting buyer. I’ve actually seen agencies drop one aging product only to purchase another - not good news when the agency realizes they’ve made a bad decision.
While the typical agency struggles with the frustrations of budget cuts and dying systems, the private sector has introduced, and now uses, newer products and technologies such as “cloud” based systems, advanced database technology such as SQL, .net and others, better statistical analysis tools, and a wide array of sophisticated handheld technologies - a myriad of newer products that would greatly improve operations in our agencies if only we had the budget to do so.
The good news is that according to most vendors and granting agencies I spoke to at the IACP Conference in San Diego, it appears we may be turning the corner. And if that is true, it is imperative that we begin planning now so that we can minimize the amount of time it will take to replace our legacy systems and make sure the decisions we make will ensure a long lifecycle in the new systems we purchase. It is also time to reflect on the history of project failures in this country and plan accordingly so we can circumvent the high-risk issues and maximize the chances for success in our next project.
Everyone knows, for example, that for decades the primary objectives for any technology project is to complete the project “on time and within budget.” Most everyone also knows, however, that those two objectives are very difficult, and in some cases, simply impossible to achieve. There are a myriad of reasons why - here are a few suggestions you may want to consider:
Completing a project “on time” is almost always impossible because when a project begins and the first project schedule is created, it is based on a limited understanding of future events. And because no one can see into the future, completing a project “on time” becomes virtually impossible if the project manager does not understand that schedules must be pliable and evolve as the project evolves. In fact, to draw a line in the sand insisting the project schedule be the #1 priority of a project is one of the main reasons some projects fail. “Put time as the priority and quality will most certainly suffer” a wise man once said. Instead, we suggest one must have a balance between time, quality and money; in many projects, all three must be at least somewhat flexible in order to achieve success. That does not mean your project should take an extraordinary amount of time but it does mean a balance must be found to ensure your project is a success.
The request for proposal (RFP) process is four decades old. It is an archaic and broken system that no longer works without significant modification. The original intent of the RFP was to ensure a fair bidding process for the vendors, minimize risk for the agency and define requirements so vendors could accurately bid on the project. It was suppose to allow an “apple-to-apples” comparison between vendor’s proposals during evaluation and allow the agency to make the best decision monetarily.
Today the RFP process is not so simple. The RFP tells them what you want; the vendor’s proposal tells you what you are going to get. In some cases, the two documents are diametrically opposed. In virtually ALL cases, there are discrepancies and differences between the RFP and the proposal that left unchecked, can lead to severe project related problems - in some cases to actual project failure.
To address this issue, we suggest that when you receive the proposals and select your three finalists, conduct a careful analysis of the differences in the RFP vs. the proposals and require all clarifications in writing. And don’t forget to require the vendors to submit their contract with their proposal - then evaluate the vendor’s contract with the same detail and intensive effort that you use to evaluate their proposal. Most agencies never review the vendor’s contract before bid award - a serious mistake that can lead to profoundly difficult negotiations. This is not something you want to miss.
Then, just before you award the bid, require the vendors to revisit their pricing to make sure it has not changed after all other issues (including contract issues) were clarified. If this is done right, only then will you know if you have a scope of work and pricing that matches your requirements and the basis for a contract that can be easily and reasonably negotiated with the vendor.
- Finally, require a strong but fair information system contract. Do not use the vendor’s contract and do not expect the vendor to simply agree to a City or County contract. Only a “hybrid” contract will work but even then the typical hybrid contract must include many very important provisions that are missing in most vendor and government contracts. Here are some examples of contract provisions you might want to include:
Defects – define Defects and Final System Acceptance in the definition section of the agreement and include provisions in the contract that require all defects to be remedied as a condition of acceptance. Vendors won’t object to fixing serious defects, and if the defect is small why would they not? Get the idea?
Interfaces – make sure you know the maintenance issues on all interfaces regardless of whether they are custom or standard. Negotiate support and fixed price costs with a cap on increases each year. If the maintenance contract is annual, make it automatically renewable at your sole and exclusive right to do so but also include a termination for convenience clause in case you want to cancel the agreement.
Final System Acceptance (FSA) – define FSA as that date when ALL terms and conditions of the agreement have been successfully completed by the vendor. Too many times agencies sign contracts where acceptance is automatically triggered by system use – something that should NEVER be acceptable to an agency especially if the investment is significant and failure is an unacceptable alternative. There are many facets to a CAD/RMS project - installation, training, software, interfaces, testing, and acceptance. Limit what the vendor has to achieve for acceptance to occur and you can be sure the system installed will only achieve that for which they are required under the contract to achieve.
Testing - recognize that testing alone is usually nothing more than a product demonstration. In most cases, it does not include interfaces or custom modifications and rarely does a vendor’s contract allow you to use the system in a live environment without accepting the product. This is one of the most serious problems facing the agency in a high cost technology project. One idea is to require all defects to be remedied and then require at least 90 days of full system operation in a live environment to prove the defects have all been fixed. Here is another thought: Don’t allow the 90 days to begin until after all contract requirements have been successfully completed. If the system does not operate for a consecutive 60-day period without reproducible defects during the 90-day system acceptance test, you may have a system that is not stable enough to accept.
- Holdback amount - typical holdback amounts are from 15-20% but rarely are they enough to keep the vendor on task to fix serious problems. Many times the vendor has simply “walked away” because it would have cost them more to fix the problems than what they would have received if they completed the project. Enter the world of the performance bond. Regardless of the amount negotiated as a holdback, if you require a 100% performance bond for the project, the vendor is sure to complete the project because losing the bond is much more severe than losing a simple holdback amount. And be sure the limitation of liability clause in the contract sets the liability to not less than the total value of the contract. Some vendor contracts will only refund the agency for the part of the system that failed - imagine getting a $30K refund on a $2.5 million dollar CAD/RMS project because they can’t get the message switch or an interface to work right. The entire system is impacted but the refund is limited to the price of the message switch or the price of the interface - a very common problem in law enforcement technology contracts.
Technology projects in public safety agencies are difficult, high-risk, resource intensive and they take time. A lot of time, planning, dedication, and attention to detail is critical; a clear strong contract will minimize the risk and help to ensure your project is a success.
Ron Jayne began his career as a Deputy Sheriff for the Alameda County Sheriff’s Department in Oakland, California. He left law enforcement to pursue a career in the public safety software first in sales at Mark Systems, Inc. and then as VP Operations for OCS Technologies, Inc. Both firms developed and sold public safety software in the US and Canada. Mr. Jayne also served as the Director of Systems and Technology for SEARCH Group, Inc., a federally funded support organization for law enforcement, and as Senior Technology Consultant for the National Center for State Courts in Williamsburg, Virginia. He is author of “Solving the Automation Nightmare in Law Enforcement” and has taught over 100 technology classes and seminars for several state and federal law enforcement associations.
If you would like help in your next high cost, high profile technology project, contact the author of this article, Ron Jayne, President of Innovative Technologies, Ltd. (ITL) at (707) 217-9766 or by email or via the ITL website.
Photos by Randall D. Larson (top: Contra Costa County Fire Dispatch, CA; middle: San Jose Police Dispatch, CA; bottom: Kissimme PD Dispatch, Florida).