The distributed control system (DCS) has been in use since the mid 1970s while the programmable logic controller (PLC) has been around slightly longer. Some early installations still are operating today using microprocessors that predate the original personal computers. The ARC Advisory Group, Dedham, Mass., estimates that over $65 billion worth of DCS installations are approaching and, in many cases, exceeding 20 years in service — and this doesn’t include other control platforms such as PLCs or supervisory control and data acquisition (SCADA) systems. So, industry faces an enormous need to replace these computer-based systems. Doing so poses a substantial challenge in justifying the investment, managing the risk, and executing the project with minimal impact to cash flow.
As with most investments, project justification for control system replacement or migration should consider two key factors: risk reduction and lost opportunity.
The risk elements associated with control systems are similar to those of many computer-based systems. An unscheduled production outage because the normally reliable control system fails can prompt an incident that poses high risks to people, production and facilities and also might lead to potential environmental violations and damage to reputation — all the typical risk matrix impact or consequence dimensions.
[callToAction ]
The main reason the control system itself fails or can’t properly be maintained is obsolescence of parts and equipment.
A related culprit is lack of people (due to retirements, cutbacks, etc.) skilled in maintaining the hardware and, perhaps more importantly, the software/applications that run the plant. Unfortunately, these legacy control systems don’t support the programming tools used today. Understanding the legacy code to ensure proper porting to the new platform can pose a real problem.
Fortunately, suitable risk management and planning can avoid this potential worst-case scenario. A relatively recent report from the International Association of Oil & Gas Producers [1] provides guidance on how to maintain integrity, high availability and low lifecycle costs in return for a small investment in planning.
The other facet of justification, opportunity lost, usually is difficult to quantify and is site specific. The opportunities generally fall into several broad categories:
• Additional capacity or production through increased asset utilization, which often is tracked as return on net assets. A control project may allow a plant to operate with reduced process deviations and, thus, closer to operating and physical process constraints without incident or upset. Modern systems use many of the same technologies found in the office environment — this significantly improves integration between the systems and, hence, visibility of plant floor data upward into the enterprise for accurate and timely decision-making.
• Improved asset utilization and lower maintenance costs. Effectively using the inherent diagnostics contained in intelligent devices able to communicate over digital networks enables better planning and also makes the maintenance exercise easier, faster and less prone to errors. These improved practices and procedures decrease maintenance costs, which may represent as much as 50–70% of the control system budget, by 25%.
• Enhanced operator effectiveness. This leads to better decision-making and fewer operational errors. Features in newer control systems aid access to information and recommended actions, including semi- or fully automated procedures or workflows that can offload many standard actions required of operators.
• Built-in cyber security measures. Newer control systems have been designed to consider and incorporate cyber-security capabilities; older systems may lack such capabilities or, as a minimum, require bolted-on security at extra cost and effort.
Justifying a project on the above factors often is challenging; it’s somewhat akin to purchasing insurance. That’s why many facilities continue to put off their migration project. Because the impact isn’t immediate, it’s an easy budget cut when money becomes scarce.
Crucial Steps
One way to prevent the perpetual delay of migration projects is to incorporate strategic planning into the annual automation system budgeting process using the information in Ref. 1 as a clear roadmap and way to document the level of risk associated with poor planning.
Once the migration activity is approved, a plant should form a project team with the objective of providing a new control system at minimal investment. Minimal investment doesn’t mean minimum cost. Because the capital investment in a control system represents at most 25% of its total lifecycle cost, the team should focus on minimum total installed cost. Moreover, with migration such a rare event, cutting corners can lead to missing some of the opportunity benefits cited above, generating a lower return on your effort. With the lifecycle of only five years for some control system components, such as servers, if a project is too spread out, the entire migration will become a continuous part of the annual budget and work plan.
Some other common “money saving” mistakes and costs associated with migrations include:
• Not following good engineering practices. Start with front end engineering design to fully understand the state of your system(s) and use an execution strategy with various steps and gates with appropriate reviews through to detail design for construction;
• Migrating the existing code directly to the new system — complete with all the dead ends that have been added through the years. This forgoes the benefits the enhanced capabilities of newer open, predominantly graphical and integrated programming languages offer as well as the use of a broader base of knowledge workers to support your system;
• Retaining the graphics from the present screens to avoid having to retrain operators. New graphics and standards present data to operators in a more meaningful way, resulting in fewer errors and, when combined with alarm management and procedural automation tools, represent an excellent way to capture the knowledge of veteran operators with little incremental effort;
• Not incorporating the capabilities of intelligent field devices into the control and asset management systems to provide notifications on the reliability of the signals upon which control is based, and using that information to minimize maintenance costs;
• Not training operations, maintenance and engineering teams on the new system(s). This not only leads to missed opportunities from your employees to make effective use of the tools available but also can actually increase facility risk, as pointed out in an accident investigation report of the U.S. Chemical Safety and Hazard Investigation Board [2];
• Not planning for the next upgrade/replacement by leaving space, real estate and infrastructure to expand the system to incorporate new capabilities or the next migration project. Some projects manage the real estate challenge by “flip-flopping,” i.e., mounting the equipment for each migration between a back plane and hinged door or side wall.
Table 1. The live cutover approach provides an attractive new alternative.
Cutover Choice
The final question to be answered — and one that significantly affects the investment profile— is how the project itself will be executed.
Migration to a new platform is a multi-faceted challenge typically tackled either wholesale during a plant outage via a “cold” cutover, or one signal at a time while the plant is operating via a “hot” cutover.
Many projects use a combination of hot and cold cutover techniques depending on the process (e.g., batch operations that regularly are idled versus continuous operations that only may shutdown every 3–7 years), criticality of the individual loops or processes, level of redundancy, and many other factors that are site specific.
One such phased approach is to replace the human/machine interface (operator stations) first so the operators become familiar with the look and feel of the new system before migration of the actual controls occurs; in other cases, the old and new systems will run side-by-side for a while.
A cold cutover also is known as a “rip and replace” project because the old equipment is removed and the new system is installed in its place during the outage. Cold cutovers also require a full plant outage to be able to cutover the common parts of the facility shared by all process units.
For this reason, many facilities use hot cutovers to migrate one signal or loop at a time by placing it in manual and then physically moving the associated wires from the old system to the new one. This obviously requires operating both systems in parallel, which demands sufficient real estate in the control room area to do this work.
Table 1 summarizes the likelihood of experiencing some of the risks associated with the different cutover strategies. As it shows, cold and hot cutovers pose different risks to production. One cold-cutover challenge not identified in the table is executing the project when everyone in the facility already is stressed to meet schedule; it also doesn’t cite the added personnel generally needed on site then and the required outage premiums being paid.
The third column covers a new migration/cutover technique [3]. This live cutover method allows for migrating control systems with minimal risk to production. It features a specialized temporary tool that enables wiring from one control system to another without disrupting the signals.
Migrate Wisely
Every facility likely will need to replace its control system at least once during its lifecycle. The challenges associated with managing an aging control system are not only how to determine the best time to make the change but also how to justify the project, execute it at minimal risk and in such a way to obtain maximum return on investment. This really doesn’t differ much from any other project; it’s just harder to visualize.
IAN VERHAPPEN is senior project manager for CIMA+, Calgary, AB. Email him at [email protected].
REFERENCES
1. “Obsolescence and Life Cycle Management for Automation Systems — Recommended Practice,” Report 551, The Intl. Assn. of Oil & Gas Producers, London (July 2016.)
2. “Pesticide Chemical Runaway Reaction Pressure Vessel Explosion,” Investigation Report 2008-08-I-WV, U.S. Chem. Safety and Hazard Investigation Board, Washington, D.C. (Jan. 2011)
3. Findlay, D., “A New Paradigm for Control System Migrations,” Control (May 2016)