Protect your plant

Production at a major U.S. chemical plant was unexpectedly shut down for two hours in March 2003, causing significant financial loss to the company. The root cause analysis indicated that the incident started when a control room operator’s computer was restarted with a changed IP address. This new IP address duplicated an address already assigned to an analyzer used for continuous emissions monitoring and the analyzer locked up as a result of network error messages. While the individual responsible for altering the IP address was never officially identified, it was reported that the address was changed so that the operators could play computer games from the control room.

This is an example of a typical security incident in the chemical processing world. There were no evil hackers involved — only employees who, while probably violating company policy, weren’t being malicious. Yet the impact from this insider threat was significant.

Interestingly, the company had a sophisticated firewall in place, based on a common strategy known as the Bastion model, where vulnerable systems are hidden behind a single firewall. Unfortunately, this design couldn’t prevent the incident because the problem originated from inside the control system, completely bypassing the firewall.

IT limitations

A number of security mistakes allowed this event to occur. First, there was an over-reliance on the Information Technology (IT) department to provide security for systems generally not in its area of expertise or under its control. IT departments are very good at providing security for systems they understand, such as Windows servers and accounts-receivable databases. Unfortunately, in most chemical companies, the critical control systems that run the processes day in and day out are strange and forbidding beasts to the IT professional.

Many process control systems have unusual operating systems and applications such as VXWorx or RSLogix that differ significantly from typical IT operating systems and applications. This means that many of the tried-and-true IT security solutions won’t function correctly or, if they do run, will interfere with the process systems.

A good example of this was reported at an ISA Industrial Security Conference in Philadelphia a few years ago. When an emergency shutdown system on a boiler failed to correctly operate, investigators discovered that anti-virus software had been installed on the computer used to configure the safety system. This software blocked the proper operation of the safety system, putting the entire plant at risk. There was nothing wrong with the safety system or the anti-virus software on their own, but together they made a life-threatening combination.

At the core, the goals of IT security differ from those of the process control world. The IT security manager sees data confidentiality as paramount (don’t let those credit card numbers be stolen) while the plant manager focuses first and foremost on human and plant safety. These differences in goals translate into huge differences in acceptable security practice. For example, using standard password lockout procedures just isn't acceptable for most operator stations in plant control rooms — default needs to be access for the operator, not lockout, the opposite of the IT assumption. Imagine the impact if, during a chemical reactor emergency, the operator panics and misspells his password three times, causing the console to lock out all access for the next 10 minutes. Password lockout is considered good policy for protecting IT servers but certainly isn’t going to work in the control room of the average chemical facility.

This isn’t to say that IT security solutions are bad for chemical processing. In fact, studies at major oil companies have shown that 90% of all IT security policies and technologies work well for industrial process control. The answer lies in clearly understanding how chemical processing assumptions and needs differ from those of the IT world and then modifying the IT security technologies and practices to properly use them in our world. This takes close cooperation and teamwork from both IT and process control staffs and not blind dependence on IT security procedures, a topic we’ll explore in more detail later.

The other mistake the chemical company made was to assume that all security problems arise from outside the plant and those that do make it in come through obvious pathways that can be managed by a firewall. This assumption often means that companies base their entire plant-floor security solution on a single firewall between the business network and the control system network, believing that their firewall will be the ultimate security filter and will prevent anything evil from ever getting to the control system. Unfortunately as this chemical company discovered, nothing could be further from the truth.