True redundancy extends beyond just the PLC to everything that could affect the controls system.
By: Allan Evora
PLC and controls only take up a small portion of an overall redundant system. Don’t get blinders on by putting a redundant PLC in your facility, only to leave instruments or networks or your SCADA system vulnerable.
Remember that true redundancy strives to eliminate single points of failure. Even if you design a redundant PLC but have a network with a single point of failure, your investment in redundancy may be somewhat diluted.
Let’s take a look at each of the five components that make up a redundant control system within a facility.
In a typical control system, you have two PLCs that do not contain any I/O modules. The controllers are strictly in place to execute control logic for the overall system. Those controllers are linked to remote I/O that reside outside the primary racks.
The PLCs are linked together via a cable which allows each controller to determine the status of the other. Only one is in control, but the other is sitting and waiting to take over control should the other have a fault. The primary difference between a non-redundant and redundant architecture is the presence of a second CPU.
Instruments are what provide feedback to the controller. For a fully redundant system, you must consider your specific application and what would happen should one of the instruments that’s considered critical toward the control of the process or equipment fail.
Temperature sensors, such as thermostats, thermistors, or RTDs are a great example of instruments. If temperature control in your facility is critical, you may want to consider redundant temperature sensors.
This mentality also applies to equipment outside of the primary control system itself, such as motors, electrical switchgear, UPS, etc.
In the event of a failure in any segment of the network, the network should continue to operate. Typically, this is done via a self-healing ring. That technology has been around for many years and many manufacturers have created their own versions.
The point is, if you’re going through the expense of designing a control system, redundancy within your network infrastructure is one of the components you need to pay attention to.
Use deterministic networks
A response from a non-deterministic network (e.g., ethernet TCPIP) is not guaranteed, which is why it is highly discouraged in redundant systems. Most redundant architectures use deterministic networks (e.g., ethernet IP and Profinet).
Here’s the reason non-deterministic networks don’t work. When performing an action on the internet at home or work, a response occurs within a timeframe that totally depends on what is happening in the network. If two people issue a command at the same time, there could be a network collision, reset, and random timeout period.
Facilities can’t develop control systems based on randomness. Consistency is paramount, which is why a deterministic network with the correct design, configuration, and protocol is important.
SCADA has many functions. It’s the operator’s HMI, historian for data logging, and also the mechanism that alerts you to the status of your high availability system. If it’s important to you that operators have visibility into and ability to interact with your control system at all times, consider specifying a redundant SCADA system.
Redundancy within SCADA has matured substantially. We like VTScada’s solution, but all major SCADA manufacturers have redundant solutions.
While commissioning isn’t part of your control system’s software or hardware, it’s a crucial aspect of redundancy. Commissioning is the formal process of point-to-point checkouts to test component functionality and operability of sequences designed into the control system.
Nowadays, most mission critical facilities employ a third-party commissioning agent or separate party who wasn’t involved in the design, development, or installation of the control system. They provide the owner piece of mind that if the system is called upon to function under one of the failure modes, it will do its job properly.
For example, we recently upgraded an EPSS SCADA system for a financial data center. The associated control system had a redundant PLC. As we went through the recommissioning process for the SCADA system, the owner took the opportunity to do a complete redundancy test. In addition to a series of tests to identify the operating and failure modes of generators and transfer switches, they conducted a full pull-the-plug test. They simulated a loss of utility power to see if the emergency generators came online, and even failed the primary controller to witness themselves that emergency power was provided to the load within the required timeframe.
They walked away confident their system would provide them with the ride-through from the generators and UPSs to allow the computing resources to continue to operate.
Moral of the story: when specifying a control system, take the time to commission that system properly.
Allan D. Evora is a leading expert in control systems integration and president of Affinity Energy with over 20 years of industry experience working in every capacity of the power automation project life cycle. With a background at Boeing Company and General Electric, Allan made the decision to establish Affinity Energy in 2002. Allan is an alumnus of Syracuse University with a B.S. in Aerospace Engineering, graduate of the NC State Energy Management program, and qualified as a Certified Measurement & Verification Professional (CMVP).
Throughout his career, Allan has demonstrated his passion for providing solutions. In 1990, he developed FIRST (Fast InfraRed Signature Technique), a preliminary design software tool used to rapidly assess rotary craft infrared signatures. In 2008, Allan was the driving force behind the development of Affinity Energy's Utilitrend; a commercially available, cloud-based utility resource trending, tracking, and reporting software.
Allan has been instrumental on large scale integration projects for utilities, universities, airports, financial institutions, medical campus utility plants, and manufacturing corporations, and has worked with SCADA systems since the early ‘90s. A passion for data acquisition, specialty networks, and custom software drives him to incorporate openness, simplicity, and integrity into every design in which he is involved.