This Month’s Puzzler
Our refinery suffered an incident that nearly led to a fire. It occurred after the board operator started the isobutanes pump on an alkylation unit surge tank (see Figure 1). The pressure safety valve popped, the flow switch in the vent tripped and so did the unit interlock on the high-high level switch.
This happened while a field operator was busy walking down one of the alkylation towers looking for leaks. That operator remembers the light on the pump station being off and assumed the low-level switch stopped the pump.
I work in a different part of the refinery and have been assigned to investigate the incident. I found the hand-operated actuator set to “off” instead of “hand” as the field operator was instructed. The display in the control room is supposed to show the pump in gray when it’s in off or hand mode to indicate that the board operator isn’t in control of the pump; the board operator doesn’t recall it being gray.
This problem involves a larger issue, though. Both the board and field operator don’t trust the automatic controls: the dP level transmitter always reads high. The board operator has compiled rough records of level based on flow meters that indicate a disparity. During the last turnaround, two years ago, the operator complained that he and the instrument technician thought the dP’s dry leg (upper) element was in the wrong place. Nothing was done then — our refinery was bought by an investment bank four years ago and maintenance has gone downhill — and the tech doesn’t work at the site anymore.
So, the board operator relies on alarms from the high and low level switches — and gets the field operator to manually shut off the pump in the case of a low-limit-level warning.
Both operators had complained to their new supervisor about the problem and were told it would be addressed during a turnaround in six weeks.
Now, rumor has it the superintendent is blaming the field operator for not being more careful.
I don’t want to rub the superintendent the wrong way; he’s politically powerful. How should I approach this problem? What do you think was the root cause of this incident?
Consider Several Potential Culprits
For the pressure safety valve (PSV) to trip, the design pressure of the vessel was exceeded, which can come from upstream valve failure at high pressure, downstream blockage assuming the pump was working, over-filling or rate of inflow not equal to outflow. I think the setting is 5 psi or 5% — whichever is higher — between the high-high pressure switch (PSHH) value and the PSV set point.
The PSHH/safety instrumented system/interlock system should have tripped the inlet emergency shutdown/shutdown valve or the high level switch (LSH) to start the pump before flow gets to a dangerous level. For both to fail at the same time raises credibility issues about the control configuration wiring and the programmable logic controller (PLC) programming.
Also, given the pump was found to be “off,” it is possible that an LSH switch/relay conveyed a message to the pump but, because it was not available, the buildup went on to pop the PSV.
A comparison between the level gauge and level transmitter (LT) would show if there are real discrepancies and if a false signal can misinform the pump. However, it is not likely in this case as the pump was reported to be in the off position. Retrain the operators and develop a standard operating procedure to forestall such occurrence in the future.
Dennis Omenka, process engineer
Petrogas Systems Engineering Ltd.,
Although saving your career should be secondary, it may be possible to do a competent, fair investigation without impaling yourself. The superintendent sounds scary but it may be that his power doesn’t extend to corporate engineering. You have to know who your friends are and who his are. By friends, I mean ones that can keep your confidences.
Your first strategic move is to get an outside team. That’s what corporate staffs are about. If possible, try to demote yourself to technical expert rather than facilitator of the team. If you’re stuck in the facilitator role, it’s imperative the team be as far away as possible from the superintendent’s influence. You don’t want spies on the team. I once got burned by a vindictive superintendent who was good friends with my boss. I couldn’t divert his wrath because I was the only one investigating the problem. Sacrificing an engineer for politics is poor corporate management but unfortunately happens.
First things first: get the unit up again. The steps are: 1) inspect the pump; 2) replace the PSV — usually it’s mandatory; 3) pull the LT — wet leg, dry leg — inspect the impulse lines for plugging, freezing, leaks, etc.; 4) pull the level switch that works with the distributed control system (DCS) — if you can; 5) test the pump logic and check the wiring — especially the DCS/PLC and hand-off-automatic (HOA) controls; 6) inspect the heat tracing on the vent; 7) check the DCS faceplate for the pump; and lastly 8) inspect the piping, relief nozzle and header, and vessel.
Schedule a drum inspection during the turnaround. Keep good records and take lots and lots of photos. Ensure all team members appear as co-authors of the report on the findings; this will help diffuse responsibility. The superintendent can’t declare war on everyone!
Document as you go is the best policy with photos: keep the originals untouched but add notes to lower-resolution pictures you can easily share via email. Add orientation pictures showing the tank and pumps and surroundings. Document as though you’re communicating with outsiders.