Case Study: Novel Approach Enhances Control System Upgrade

Figure 1. Project activities took place over a two-year period.Software. The software team comprised two process control engineers from Lilly’s central corporate engineering tech center who were familiar with DeltaV, one plant process control engineer who knew the process and equipment being converted, and several contract system integrators. All assigned internal resources were dedicated to the project and were very experienced with either the existing Measurex DCS and its current operation or the functionality of the new DeltaV DCS. The selected system integrators previously had worked either in the plant being converted or on other corporate DCS projects. The software team prototyped the proposed design; the aim was to be consistent with ISA S88.01 [1] and include all documentation outlined by GAMP [2]. The prototype helped minimize rework by allowing operations, engineering, quality control and the software team to agree on an initial design before large amounts of work were expended. This initial prototype wasn’t a 1:1 conversion of the existing Measurex DCS software. While it was important to retain the knowledge built into the system over the years, it was equally important not to squander opportunities for improvement afforded by implementing a modern DCS. To avoid introducing unintended risks during startup, only improvements that wouldn’t affect the quality of the API were made. For example, when automating a pH adjustment, the team designed an algorithm that mimicked current operator technique. We used many sources, not just the existing Measurex DCS code, to generate the first pass of the requirements and design for the new DeltaV DCS. Other inputs included existing software requirements and design documentation, process historian data, standard operating procedures (SOP) and manufacturing tickets. The process team, consisting of representatives from operations, process engineering and tech service, as well as a dedicated senior operator, then reviewed this documentation in great detail. Once the process team approved the documentation, coding began. The requirements, design documentation and coding standards all were very specific. This specificity resulted in very few surprises with the initial software delivered. The plant automation engineer on the team assessed this software using a process simulator and suggested some changes. Once these were made, the process team examined the software via the simulator. Following this review, the software was formally tested with the process simulator. This served to minimize any surprises during commissioning and qualification.

Figure 2. Creating a replica of the actual I/O room paid off in reduced hardware installation costs and shorter shutdown time.Hardware. With the scope of the hardware portion of the project effectively limited by the “Just Say No List,” the hardware team verified the landing points of all existing I/O, documenting results in a controlled project I/O database. All I/O for this particular Class I, Division 2 control room was run from the field into a central secured I/O room above the dedicated operator workstation area. The general hardware strategy was to remove the Measurex DCS field termination boards, control modules and computer system, terminate the wires onto generic termination modules and then connect these modules to the new DeltaV DCS termination boards and controllers using multi-pin connectors whenever possible. Operations elsewhere in the building required the purified water handled by this control room well before the project would be completed; so the team developed a different strategy for the 230 purified-water I/O. New wires were run from the field devices to a prefabricated field panel. This setup allowed electricians to simply swap the wires at each device as soon as the shutdown commenced without having to terminate the wires in the panel. In an effort to minimize shutdown duration for the remaining I/O points, we built a replica of the I/O room, known as the mock-up room, in an adjacent basement — this level of physical simulation of an I/O room was new to Lilly DCS upgrade projects. The mock-up room (Figure 2) enabled electricians to complete some work in advance of the shutdown (e.g., cutting wires to length, pre-labeling terminations, and fabricating controller and termination board panels) while getting a feel for working conditions during the shutdown. The mock-up room also allowed the hardware team to predict with a great deal of certainty the amount of time needed by electricians during the actual shutdown. This led to a shorter shutdown — the money saved by this more than paid for the cost of creating the mock-up room. New operator workstations, controllers, servers and two fiber optic networks also were needed. The operator workstations essentially were replaced on a 1:1 basis. We determined the number of new controllers and servers based on past experiences with DeltaV at other similar facilities. Operator training. We used two primary methods to get operators up-to-speed on the new DCS. The first involved a short computer-based training course developed in house to cover the basics of the new control system (navigation, alarming, graphics, logging in, etc.). Operators completed this training in advance of the shutdown. Once the shutdown started and removal of the old control system began, operators attended roughly 40 hours of training courses that used a process simulator that exactly replicated both the hardware and software of the production DeltaV DCS. The latter courses comprised an initial walkthrough of the new software followed by simulation of production batches using manufacturing tickets and SOP. These simulations included non-routine events, to give operators experience in this type of event handling. Commissioning and qualification. Upon full wiring termination and power-up of the new controllers, loop checking began. The C&Q team developed a refined loop-check strategy to minimize cost and schedule impact. The nature of the conversion project meant the majority of the field wiring remained intact. So, the loop-check procedures allowed for testing only wiring impacted by the upgrade. We tested analog instruments just for signal continuity rather than the traditional multipoint current simulation at the field device. Loop checks for automated valves were completed automatically — a custom program made each valve stroke open and close. Any valve state failures received additional attention using more-standard loop troubleshooting techniques. Thanks to this approach, we verified proper wiring on 200 of the 250 automated valves in 20 minutes without any manual labor [3]. With the loop checks completed, C&Q commenced. The C&Q strategy typically involved running a water (or process buffer) batch to test code functionality, tune control loops and check batch totalizers. The water batching also allowed for in-process leak checking, minimizing health and safety issues associated with restarting the equipment. Operations personnel executed the water batches, with process engineering, automation and contract C&Q personnel on shift. The familiarity the process team and operations personnel already had developed with the software led to a relatively uneventful qualification.

Figure 3. The first upgrade laid the groundwork for future projects and thus incurred higher project management costs.The ResultsWe met the primary goal of the project — safely upgrading the DCS in a timely and cost-effective manner. The shutdown duration was shorter and the total cost was lower than for similar projects within the company. We kept total cost well under budget and also achieved benefits in alarm rate, cycle time and number of manual actions required. Alarm rate was cut by 95% [4], while cycle time of the bottleneck step in the control room was reduced by 6%. In addition, we automated numerous manual activities. For instance, passing data automatically from one unit operation to the next removed calculations from manufacturing tickets. In effect, tanks determine their own addition/dilution volumes, columns ascertain their own charge volumes, etc. Buffer operations were automated to the extent that SOP no longer were needed — representing a 40% cut in the total number required. Also, we automated many non-routine operations (special cleanings, purified water sanitizations, etc.). The conversion included equipment state tracking (hold clean, run counts, etc.), as well. Given the success with the first control room, Lilly decided to upgrade two more control rooms in late 2008. Using the methods developed during the first upgrade, this follow-up project provided even better results. It achieved a 36% reduction in cost per I/O, while attaining similar benefits to those observed in the first upgrade. Figure 3 shows the cost breakdown for both the initial and follow-up project. As the second upgrade was completed, another Lilly API facility started a similar Measurex-to-DeltaV project, with implementation scheduled for mid-2010. AcknowledgementThe author thanks Ben Paterson, Kevin Duffy, Kevin Wilhelm, Brian Hrankowsky, David Chan, James Cox, Eric Hubert, Brett Conaway (ACE Technologies, LLC), Graham Smith (Brillig Systems Inc.), Doina Morusca (Invensys Process Systems), and the Cornerstone Controls Inc. team for their contributions to this article.Ryan M. Iannucci is a senior process control engineer for Eli Lilly and Co., Indianapolis, Ind. E-mail him at[email protected].References1. “Batch Control Part 1: Models and Terminology,” ANSI/ISA-S88.01-1995, The International Society of Automation, Research Triangle Park, N.C. (1995).2. “GAMP Guide for Validation of Automated Systems: GAMP 4,” International Society for Pharmaceutical Engineering, Tampa, Fla. (2001).3. Chan, D. and B. Hrankowsky, “Migration of a Biotech Insulin Manufacturing Plant from Measurex to DeltaV,” presented at Emerson Global Users Exchange, Washington D.C. (Oct. 2008).4. Iannucci, R., “A Practical Approach to Alarm Management,” presented at ISA Expo, Houston, Texas (Oct. 2008).