Most hardware failures in the Bailey Network 90/Infi 90 DCS were in the DEC Alpha-based, OSI Conductor VMS consoles that had replaced original Bailey MCS consoles. DEC parts are out of production; used parts are difficult to locate. We also were seeing occasional failures in DCS controller input/output (I/O). In addition, an associated programmable logic controller (PLC) running the Lima site’s deep-well injection system was suffering spurious failures of unknown origin that tended to be in attention-getting critical logic and EPA-required recordkeeping. Opinion was divided on whether the failures were random or symptomatic of equipment age and possible failure acceleration. We initiated an analysis to find out but results were inconclusive.
The next scheduled turnaround for the acrylonitrile unit isn’t until 2010, so management had to grapple with whether the plant could wait four years for an automation upgrade or replacement. The decision was that it couldn’t.
Upgrade or replace?
We evaluated three potential solutions:
- Refurbish the VMS console computers, replace the lower CRT monitors with LCDs (the top CRTs already had been swapped out as a stopgap), and hope for the best from the controller hardware until 2010. This option posed two major downsides: the cost and questionable quality of refurbished computers to gain only a three-to-four-year reprieve; and, of course, potential controller and I/O unreliability.
- Replace the old console computers with new HMI hardware and software from the same company. The downsides: the difficulty in reconfiguring from Conductor VMS to Windows; time-consuming graphics conversion because no automated utilities were available; doubt about whether the consoles could be replaced without shutting down acrylonitrile production; and, again, questionable controller and I/O reliability.
- Install a new DCS, plus perhaps a new PLC, as soon as possible, with cutover of the acrylonitrile operation accomplished hot. The major downside here was a possible cutover-induced process trip.
None of these options involved changing field devices or altering control strategies.
We found that the last alternative was the only practical choice. First, the cost of configuring and switching from existing consoles to new workstations was lower than that for retaining the old HMIs. The new DCS also offered the best avenue for hot cutover of both consoles and reactors — one reason is that it incorporates advanced digital automation, so we could avoid the need for add-on technology and the attendant chance of mistakes. Last, Ineos has had excellent experience with this automation equipment at other sites, including an adjacent catalysts plant.
However, management was very concerned about a hot cutover; no one at Ineos to our knowledge had ever retrofitted controls for such a large and important process while it ran. Management also questioned whether enough real estate existed in electrical equipment rooms for both new and old control cabinets to run simultaneously and thereby allow a hot cutover. We won over management by developing a two-phase, six-month plan. This involved replacing the HMIs first, then the controllers, and finally the I/O.
The plan
The first phase employed Emerson’s DeltaV Connect Solution for Bailey Systems, a transition package that replaces the existing HMIs and provides modern functions, while retaining the original system’s controllers, I/O and field devices. The new PC workstations were mounted in semicircular console furniture (Figure 1) and connected in parallel with the existing consoles, which were moved against the back wall of the control room and remained live and operable until the new workstations were tested and proven to work with the existing controllers. To operators, the transition was essentially seamless; I don’t believe any of them ever used the old HMIs after the new workstations were up and running.
The transition package seamlessly transferred the original DCS’ analog and discrete points plus loops into the interim workstations, as well as node operating status, configuration information and online tuning. No reconfiguration of old equipment was required. When connected in parallel, operating and tuning changes made from the transition package were automatically updated into the original system and vice versa. Very helpfully, the transition package added state-of-the-art event reporting, history collection, and enhanced alarming and diagnostics to the original controls.
At operator request, the original graphics, including alarms, were replicated as closely as possible in the transition package. Graphics followed guidelines of the Engineering Equipment & Materials Users’ Association (EEMUA) Publication 201 with regard to alarm management, colors, contrast, etc.
Conversion from old consoles to new workstations took less than one week per control house.
Phase 1 also included paralleling and then switching serial links from the old equipment directly to the project’s first new controller. This avoided two-step conversion (old to interim to new control) of 1,400 serially communicated points.
All Phase 1 work was accomplished hot with the plant running. No process trips occurred.
Phase 2 replaced all the old controllers and I/O equipment, as well as the deep-well-injection/surge-tank Modicon PLC, with new controllers, I/O, workstations and networks. The interim workstations were slowly converted to pure DeltaVs.
Cutover
The cutover of the deep-well/surge PLC and the acetonitrile unit was next on the list. We performed a cold cutover because this work could be scheduled so it didn’t impede critical acrylonitrile production.
The PLC had 148 points in the pump building’s main rack for injection logic and another 60 points for surge control in a coax-connected remote I/O rack in a distant control cabinet. The conversion of both could take no more than three days — the time it would take for the surge tanks to become full. Because space was available in the injection building, a new controller cabinet was installed near the PLC cabinet. Cutover required removing the PLC and then both retraining existing wiring and adding wiring from the old cabinet to the new. The old cabinet remains as a marshalling enclosure. The two-stage PLC conversion task was completed in three 14-hour days.
Cold cutover of the acetonitrile unit was scheduled during a five-day outage that also included non-control work. The switch involved straightforward tearing out old cabinets, installing new cabinets, wiring up and checking out 292 I/O points, about 75% analog. Cutover took four days, plus a fifth day for tuning and tweaking.
It was important that a few parameter indications, such as cyanide drum levels, be relayed to the control room during cutover. This was achieved by temporarily powering the new acetonitrile cabinets with an extension cord to limit the interruption of vital indications to a matter of minutes.
The acrylonitrile unit was cut over last and hot. The cutover of its 1,023 I/O points was spread over a two-month period ending in late November 2006. A team of two experienced contract electricians, a very knowledgeable and helpful operator, and me worked full-time on the project. The operator was essential for knowing how the control strategy and loops work — or should work — and for performing all of the necessary bypassing and hand-jacking of control valves. Although I know the process quite well, I’m a controls engineer, not a process engineer.
On average, we converted and tuned three to five active control loops a day. Indication points usually were converted and rung out while waiting for loop tuning and process bumps to settle. The original tuning methods and parameters generally translated well. We made rough calculations of the tuning sets in the old controllers and then plugged these parameters into the new system. Many of the loops were also then tweaked using DeltaV supplied software. Our tuning upsets made operators nervous at times but never affected product, equipment or output.
Much of the hot cutover was physically aided by hand jacks and bypasses. I’ve come to believe that every regulating valve should have a bypass if safety isn’t an issue. Hand jacks require too much care with settings to avoid the valve becoming fully opened or fully closed.
We accomplished the acrylonitrile unit cutover without a process trip. Unfortunately, failure of an associated ac power device due to a wiring error made after cutover did cause a trip, but this wasn’t attributable to the automation or its cutover.
Only one element of the former controls remained: a standalone PLC running truck and railcar loading. It was served by two HMIs in a setup that wouldn’t allow the operator to shift from one PC to the other once loading started. These controls were replaced at the end of the project with a more efficiently programmed HMI that communicates with the PLC over Modbus. This improvement will allow for loading to be controlled from anywhere on the system’s network.
Significant benefits
The hot cutover provided substantial savings. At our plant, the cost per day in lost acrylonitrile production is roughly $100,000 at today’s product value. A simple trip requires a 1.5-day restart.
In addition, we’re gained a number of advantages from the new system:
- There haven’t been any control failures due to the automation since conversion.
- The plant turnaround in 2010 won’t be complicated or extended by the installation and startup of a new automation system.
- The new system’s scalability and flexibility allows us to relatively easily add, enhance and change automation hardware, software and strategies to suit operations. One cabinet has already been fitted with a Modbus TCP/IP card for the fastest communications with Modbus devices. All cabinets have plenty of room for expansion with 20% spare conventional I/O capacity (Figure 2). Every new cabinet has a Foundation Fieldbus card and power conditioner even though no fieldbus has been installed yet. The adjacent Ineos Catalysts plant, also under new automation, relies upon field devices very successfully served by Foundation Fieldbus, AS-Interface bus and Profibus DP segments. So, we aim to employ bus connections in the future in our units.
- We finally are making use of data from HART instrumentation that’s been in place for years. Especially helpful have been secondary variables from Coriolis mass flow meters, plus indications of oscillations in some valves, thanks to position feedback that couldn’t be sensed before.
- The DeltaV Analyze tool is helping to identify nuisance alarms, and the system is enabling us to quash such alarms by suppressing a particular parameter alone, not the entire tag. For instance, we temporarily quashed a high alarm that was a nuisance due to a field issue without suppressing a critical low alarm; this wasn’t possible with the old system where suppression took place at the tag level instead of at the individual alarm level. We’ve also added alarm suppress/re-enable when certain equipment is out of service. A pop-up faceplate allows operators to auto-suppress hand-chosen alarms via calculation blocks built into control modules. Hysteresis adjustments have permitted us to provide a larger dead band on some alarms, which also has eliminated some nuisance alarms.
The system promises additional benefits in the future. For instance, we’re considering eliminating an AspenTech DMC platform on the acetonitrile unit and incorporating its functions within a DeltaV PredictPro model-predictive-control package. This would cut the costs of five PCs and an annual maintenance contract.
We’re also considering opting for a safety instrumented system (SIS) integrated with the DCS for new construction. Our Triconex system works very well but is a separate platform that required substantial serial communications configuration into the new system during the recent conversion. A new SIS package is working well at the Ineos Catalyst plant for burner management; I’ve been told the added diagnostics are quite helpful.
A better approach
Looking back over our conversion experience, I don’t think I’d ever want to convert a process automation system for a critical unit as large as ours if it were not running. Trying to start up, troubleshoot and tune hundreds of loops all at once would be a nightmare.
There are other advantages to our approach as well. Separating a hot conversion effort into HMI and controller-I/O phases enables operations and maintenance staff to become comfortable with the replacement automation’s HMI without concern about controllers and I/O. Personnel prefer to adjust in small bites to many minor graphical differences. Typical might be the new system’s Auto and Cascade positions for switches versus the old system’s Auto and Manual. Later, personnel can focus on process logic changes while controllers and I/O are converted without, at the same time, being overwhelmed by a new HMI.
Eric Schnipke is a controls engineer for Ineos Nitriles, Lima, Ohio. E-mail him at [email protected].
