Many chemical plants now need to upgrade or replace their aging process control systems. Modernization offers substantial benefits — such as lowering costs, increasing control system effectiveness as well as plant performance and flexibility, generating real-time business intelligence from operational data, and easing regulatory compliance. However, it also raises significant issues, including security and regulatory-compliance ones.
Five major hurdles to achieving and maintaining security and compliance are:
1. Lack of "last mile" coverage and instrumentation for device visibility;
2. Not so automatic "automation;"
3. Data overload;
4. Inability to detect anomalous behavior; and
5. Collection, analysis, and workflow lifecycle integration.
In this article, we’ll look at what you can do to overcome these hurdles.
DEVICE INVISIBILITY
Plant control systems increasingly are leveraging wireless and business connectivity to expand their reach and effectiveness. Gaining faster access to more granular and real-time data from remote endpoints can produce substantial operational benefits. However, from a security perspective such expansion introduces new risks.
One of the primary security issues is posed by intelligent endpoints such as programmable logic controllers (PLCs) and remote terminal units. Because these lack local or remote logging capabilities, they can’t adequately log relevant security and operational events. However, plants covered by the U.S.’s Chemical Facility Anti-Terrorism Standards (CFATS) must collect data, such as events and configuration details, to adhere to these standards. Furthermore, interactive remote access to these endpoints can be cumbersome, hard to achieve or only available in an insecure manner.
What you can do: To address the lack of visibility into these devices, consider placing network sensors near them in the control system to detect events that normally would appear in event logs. Network intrusion detection systems (NIDSs) and network flow tools are two options that provide potential workarounds for this lack of endpoint visibility. NIDSs effectively monitor network traffic to and from endpoints. NIDS devices can be configured to trigger on events such as when shutdown or reset commands are sent to PLCs or when privileged-user accounts are logging into them. Most NIDSs also enable users to create customized rules to accommodate unique plant requirements. Moreover, they usually allow signatures to be built around the Modbus, ICCP and DNP3 protocols common in industrial process environments.
Additionally, because many of the industrial protocols used in chemical plants today lack solid authentication and security features, consider protocol-aware gateways or firewalls to restrict access and add another layer of security.
NOT SO AUTOMATIC "AUTOMATION"
Plant managers also are facing growing internal and external (regulatory) mandates, such as CFATS, and either already are or soon will be required to produce and report on enormous amounts of data. So, they must find an efficient and secure way to deal with this growing operational and administrative burden. Compounding the problem, sites often have hybrid environments with multiple control systems from different vendors. Each of these systems may provide its own point solution to help address particular aspects of these requirements. This results in a patchwork of systems and functionality with overlaps that make system management difficult and confusing.
Managing multiple point solutions is a suboptimal approach. Instead of simplifying operations, it complicates matters and increases administrative overhead. For these reasons, many plant managers resist fully automating their monitoring and data collection processes, or simply can’t get an automation project started because they lack the resources or expertise to handle the additional overhead.
This inability to fully automate data collection efforts often leads to partial automation efforts. Examples include manually running scripts on each individual host or remotely running scripts that have to be manually initiated. These half-measures are neither thorough nor rigorous and typically yield incomplete results. Moreover, lack of experience in writing scripts potentially may pose risks to the availability of control systems. Reliance on such manual scripts is a common problem that prompts the adoption of unsustainable processes and robs engineers of time better spent on running and optimizing plant operations.
What you can do: Many solutions for automating data collection processes safely, securely and effectively are available on the market. By embracing a fully automated approach to this increasingly strategic activity, plant managers can safely meet their data collection and reporting requirements. Full automation also delivers the benefit of greatly reducing or eliminating tedious time-consuming, expensive and error-prone manual processes.
It’s important to understand that automated data collection isn’t the same as "network scanning." Automated data collection takes advantage of a control system’s built-in administrative capabilities, collecting data in a controlled manner while creating very little overhead on the endpoints. In contrast, network scanning is associated with network-based port scanning, which, if not done carefully, can affect the availability of the control system.
DATA OVERLOAD
Frequently, raw output from tools used to collect security and compliance data is all encompassing and complete. That’s the good news. The bad news is that this usually creates an overwhelming amount of data that requires specialized knowledge to understand. One common example of the data overload problem is log data that are dutifully collected and carefully stored in a repository but then never looked at nor analyzed. Understanding the data is one thing but finding that proverbial "needle in the haystack" is another. Process engineers may know the systems very well but interpreting the log output from a variety of device types can pose a real challenge. Gaining insight and business value from log files often requires specialized knowledge and experience, especially when it comes to correlating events. If plant staff doesn’t possess the requisite skills, the data collection exercise can wind up an enormous waste of time and money.
What you can do: Fortunately, solutions that automate data collection and also provide correlation capabilities are available — enabling staff to easily and quickly find the events of interest to them, regardless of the endpoint type, without having to wade through mountains of raw data.
Without the right tools in place, a plant manager may lack complete information and, thus, may get an inaccurate picture of the site’s security and compliance state — and that obviously can precipitate other, bigger problems. Using the appropriate solutions designed for plant environments can greatly help a manager make operational assessments with much greater visibility, accuracy and completeness.
NOT DETECTING ANOMALIES
Closely related to the data overload issue is the inability to spot atypical activities occurring on control systems and network segments. Zero-day attacks (i.e., ones that target system vulnerabilities that are unknown at the time of the attack) can devastate control systems. Because there’s no patch or fix at the ready, great damage often can result.
Many people underestimate the lag time between the launch of an attack and when the patch to the control system is completed. For example, an unknown vulnerability can exist in business and plant environments for several weeks, months or even years before it’s noticed. Once the vulnerability has been discovered, the vendor of the operating system or application should create and release a patch within days or weeks — although this sometimes takes months or longer. Then, the control system vendor must test and approve the patch or hot fix for its system, which can incur a significant additional delay. And finally, once the patch reaches the plant, installation often has to wait due to availability requirements. If plant managers aren’t sweating during every hour, day and week that passes during these often-extended periods, they should be because their critical control systems potentially are exposed and vulnerable.
A prime example of this was the Nitro attacks, which are thought to have started in early 2011 but weren’t discovered until late that year. They targeted 29 companies in the chemical sector alone, and appeared to be an effort to steal intellectual property. Attacks such as these are known as advanced persistent threats or APTs. They are designed to be very stealthy as they slowly accomplish what they’re intended to do. APTs generally are nation-state sponsored and often target industrial environments such as chemical plants. They are extremely difficult to detect.
What you can do: A great tip for helping to detect these types of attacks is to ensure the continuous monitoring of the plant environment, including the control systems and networks. Monitoring not just security and operational events but also the configurations on each endpoint enables establishing a baseline for normal activity. When an activity out of the norm occurs, staff can be quickly alerted to determine its nature. This is similar to exception-based alerting and reporting, and is a great tool for identifying changes in the plant environment. Exception-based tools also are generally automated and designed to reduce the resource overhead for day-to-day security and compliance requirements.
WORKFLOW LIFECYCLE INTEGRATION
Many plant managers will stop at the collection step and declare their security and compliance efforts a success. However, data collection is just the first step. To be truly successful, plant personnel must be able to collect, analyze and then act on the security and compliance data that have been gathered. By continually iterating over and acting upon the data, a site can track and improve its security and compliance efforts over time.
For example, logging failed logons provides no value by itself. Without an analysis of these events, the plant won’t know of the failed attempts. In addition, if the failures are malicious or the events are from a service configured to use an expired password, they could indicate potential availability issues with the application.
Similarly, just logging events to meet a compliance requirement won’t suffice. How will someone know when log data collection fails or if there’s a gap in the collection? Without tracking the dates, times and failures of log collection, the plant leaves itself vulnerable to a compliance deficiency.
What you can do: Rely on automation for lifecycle integration. Without automation, the data collection, the data analysis and the response processes quickly become unmanageable and unsustainable. All too often, they’re done on a manual and "as needed" basis, if at all, and generally must be repeated in another six to twelve months — giving the plant manager only a snapshot in time of the status of the plant.
THINK DIFFERENTLY
Old approaches to control system design and security are becoming increasingly ineffective in the face of major technology trends and business changes. Forward-thinking plant managers must find effective ways to overcome these new security and operational challenges.
The first step is recognizing that, in many areas of plant security, what has worked in the past likely won’t work in the future. So, it’s crucial to explore new options and develop effective business cases for investing in next-generation plant security technologies. By embracing the changes taking place in the chemical industry and adopting new solutions to address them, plant managers will be able to mitigate risks and capitalize on the terrific opportunities that lie ahead.
JACOB KITCHEL is senior manager of security and compliance for Industrial Defender, Foxborough, Mass. MICHAEL PICCALO is director of customer solutions for Industrial Defender. E-mail them at [email protected]
and [email protected].
