Is your SCADA system running on a non-supported OS? Are your PLCs obsolete?
By: Allan Evora
One of the biggest challenges for some of our emergency power system owner-customers is that systems installed 10 years ago are running on obsoleted operating systems (OS) or currently not supported by Microsoft. Specifically, Windows XP and Server 2003.
Recently, we worked on an EPMS system recently refreshed in 2014, and for the last 3 years had no Microsoft updates or patches applied. The company was found out of compliance in an internal audit, and we were contracted to quickly bring the computer up to date.
Thankfully, owners are finally starting to think about what OS end of life means for their Supervisory Control and Data Acquisition (SCADA) systems.
Why so many recent upgrade requests?
In the past year, we’ve had numerous requests for computer upgrades. Here are a few reasons why.
Part of this sudden interest in outdated operational systems that had previously flown under the radar is due to the criticalness of widespread threats, such as WannaCry and Petya ransomware. Today’s threats are spread through multiple mechanisms. Just because a computer or control system isn’t connected to the internet doesn’t mean it can be left without updates that address security vulnerabilities.
In fact, these threats are so critical that Microsoft has actually released security updates for older, unsupported operating systems like Windows XP and Windows Server 2003 even though Microsoft discontinued support for both operating systems in April of 2014 and July of 2015, respectively.
Automatic patches don’t work anymore
In older, unsupported operating systems, the mechanisms in which patches are applied can be significantly different than a simple automatic update within the OS. They often have to be manually applied, which is not something many facility managers know how to do.
Several owners of emergency power supply systems have asked us to regularly visit and perform IT maintenance on their systems, including the regular release of manual security patches to address threats.
IT departments are conducting more scrutinizing audits on computer assets throughout the entire organization’s premises. Most SCADA or controls systems have flown under the radar for so long because they were developed in isolation from the IT group. This is changing.
If an emergency power system computer hasn’t been maintained, has an older OS, or hasn’t been getting its updates, it could represent risk…even if it isn’t connected to the network.
For example, we’ve had customers who built backdoors into their SCADA system, or who connect to the building’s guest WIFI and use GoToMeeting to enable vendors to support their systems. After an audit, both these practices were shut down immediately due to the high risk they posed.
Think beyond the OS while planning upgrades and updates
Updating or upgrading is part of the cost of owning a system. Maintaining the operating system and its different updates is part of normal preventative maintenance routine.
When considering a significant upgrade to a computer and associated OS, we recommend owners not just think about the computer that hosts the SCADA software, but also the actual SCADA software, controls hardware, associated HMI, and recommissioning.
Upgrading the OS might also require an upgrade to the SCADA software. SCADA software – if developed at the same time as the OS implementation– will most likely not be forward compatible with the new operating system.
Several popular original equipment manufacturers (OEM) are going through product obsolescence, like GE Series 90 PLCs. Although parts can be obtained through a third party, they’re likely refurbished or extremely limited (and therefore very expensive).
When looking at total cost, it’s easier to explore an upgrade to the latest version of the control platform. Luckily, upgrading GE Series 9030 to the current platform (GE RX3i) is a very straightforward process. It doesn’t require any physical modification like wiring and harnesses, and the platform itself is easily migrated.
PLCs are most often connected to a local HMI or operator interface terminal. If you’re considering a PLC upgrade, the HMI is a critical piece that shouldn’t be overlooked. What if a replacement part is no longer available? At the very least, it’s certainly worth considering the “what would we do if this HMI fails” scenario.
We recommend a recommissioning or reverification test be performed whenever a significant upgrade of SCADA, computer, software, or PLC controls and associated programming happens.
Find a control system integrator with the necessary IT experience
Typically, a control system integrator like Affinity Energy can offer upgrades, especially for local customers, at a significant cost savings over the OEM.
Being an integrator with extensive experience in a wealth of different PLC, SCADA, and software manufacturers means we can incorporate an upgrade program as part of our regular preventative maintenance program.
A lot of our customers ask us to upgrade their computers, then visit the site on a quarterly basis and perform backups, determine if a new disaster recovery image is needed, archive data, apply security or system patches, and create reports that show the patch was successfully applied.
This IT-type skillset is something most control system integrators may not have internally.
As part of our planned visits, we also try to incorporate some basic checks of the SCADA and control system and talk with the operators to see if there are any areas that require attention, for example, a recurring nuisance alarm, or an instrument that appears to be going out of calibration.
Lowering the overall costs of labor during patch updates
One of the benefits Affinity Energy can bring to the table is a more effective way to apply patches.
Downloading patches at the customer site can be very time consuming. In addition, some facility computers aren’t connected to the internet, or the process to get a connection is difficult. One of our customers with six view nodes has to put in an IT request each time they want to open a port on their computer.
It all represents time that the customer ultimately pays for.
Instead, we download patches and updates before visiting the facility so we don’t waste a customer’s time sitting around waiting for file downloads. We don’t require internet access onsite, which means we can apply security updates with minimal impact.
Update and upgrade on your own terms
Facility managers must realize the importance of OS upgrades, and the associated upgrades to their SCADA and other instrumentation and controls. If systems go down or aren’t being maintained and cause an event, accountability lies with them. They can’t continue to operate as an island.
I like to see owners upgrade and update on their own terms, not because of a system failure. It’s always easier to take a preemptive approach when planning out upgrades, especially when considering the impact of downtime and loss of visibility to various equipment.
Need IT help from a control systems integrator who understands emergency power control systems? Let us help.
Allan D. Evora is a leading expert in control systems integration and president of Affinity Energy with over 20 years of industry experience working in every capacity of the power automation project life cycle. With a background at Boeing Company and General Electric, Allan made the decision to establish Affinity Energy in 2002. Allan is an alumnus of Syracuse University with a B.S. in Aerospace Engineering, graduate of the NC State Energy Management program, and qualified as a Certified Measurement & Verification Professional (CMVP).
Throughout his career, Allan has demonstrated his passion for providing solutions. In 1990, he developed FIRST (Fast InfraRed Signature Technique), a preliminary design software tool used to rapidly assess rotary craft infrared signatures. In 2008, Allan was the driving force behind the development of Affinity Energy's Utilitrend; a commercially available, cloud-based utility resource trending, tracking, and reporting software.
Allan has been instrumental on large scale integration projects for utilities, universities, airports, financial institutions, medical campus utility plants, and manufacturing corporations, and has worked with SCADA systems since the early ‘90s. A passion for data acquisition, specialty networks, and custom software drives him to incorporate openness, simplicity, and integrity into every design in which he is involved.