Lessons from the Pinellas County, FL Hack

The biggest challenge the 55,000 water treatment plants in the US face is monitoring the plant to avoid intrusions. In our increasingly digital world, the threat of intrusion is no longer just physical.

Water treatment plants are amongst the focus of cyberattacks from international hackers and extremists. The recent hack of the treatment facility in Pinellas County, Florida, shows us that even the smallest operations are at risk.

Why? The reason is simple; each of these plants provides water to an average of about 50,000 Americans. The potential impact of a hack is substantial for those seeking to do maximum damage.

This figure is attraction-seeking, and the US water supply technology has very little protection against hacking.

The Florida incident is a case study to explore the prevailing problems with the existing water system management technologies in the country and how operators can better protect their critical infrastructure going forward.

BACKGROUND: The Florida Water System Hack

In the early days of February 2021, an unknown hacker recently infiltrated a computer for the water system and briefly increased the amount of sodium hydroxide by a factor of more than 100.

The introduction of sodium hydroxide would have led to the loss of disinfectant in the water, a sudden change in water PH level, and dissolved oxygen in suspension.

According to the CDC, this would have exposed residents to several health issues that may cause the skin tissues to receive chemical burns.

Though the attack was identified and averted quickly by plant operators on duty, the incident seemed to have confirmed some tech individuals’ speculations; previous suspicious incidents were chalked up to mechanical or procedural errors.

Before now, we have had cases of individuals discrediting the security systems of most public utilities facilities.

The Florida water system hack has proven that our water systems, like other facilities, are vulnerable to extremists or hackers' attacks. Hence, the need to step up the security systems in these facilities. 

The Potential Severity of the Attack

The consequences of this attack are many, but three main that stand out are: 

1. The health and safety of customers
2. The operational impact (time and money) to recover from such an event
3. The long-term damage to public trust 

Anytime a foreign or damaging chemical is introduced into a domestic water source, several things will happen. Loss of disinfectant  (Chlorine Residual drop), pH levels in the water will change (Lye being a strong Alkaline with a 14 pH), Dissolved Oxygen (DO) in suspension will also drop. 

If the lye had not stopped from being dosed into the system, the water pH level would have become alkalotic, with the pH of lye being 14 on the pH scale.  A normal pH in the human body is 7% pH. The high pH would have caused exposed skin tissue to receive chemical burns. One sip of water and the inside tissue of your mouth would have been affected by a chemical burn. 

Science tells us it takes a 10,000 to 1 ratio to raise or lower a pH by 0.1%. That is 10,000 gallons of water to 1 gallon of Lye ratio of water to lower pH by 0.1 %. 

Within a water distribution system, with this scenario, the entire production and distribution systems would need to be drained and flushed, wasting hundreds of thousands of gallons of water, costing thousands of dollars of water loss. 

What Are The Problems of Existing Water Treatment Systems

Water supplies in the United States are among the safest in the world. However, even in the US, water systems face several challenges. The consequences can range from operational disruptions to potentially putting citizens' lives at risk. 

Here are some of the problems facing existing water treatment systems:

#1. Contamination

This is, by far, one of the significant problems facing existing water systems. Contamination may be due to waterborne germs or the result of an intrusion, just like the Florida incident.

The United States Environmental Protection Agency has set legal limits on over 90 contaminants in drinking water, but many water supplies struggle to meet it with available technology.

The truth is, many of these systems are unattended or underfunded. However, with an alarm system like RACO, water plants can monitor water quality before consumption.

#2. Cyber Security

Most water systems in the US use a 'human-machine interface' or HMI to monitor and control water treatment, distribution, and storage. 

What happens if an intruder gets access to this interface?

If it’s secure enough, everyone is safe. Otherwise, the result can be disastrous.

A recent report shows the Florida plant used an old version of Microsoft Windows, and all the computers shared a single password. And these highlight evidence of poor cybersecurity hygiene at the plant.

Though the Florida system attack consequences were mild, it would have been a different story if this illegal intrusion occurred and there wasn’t a human actively monitoring the system. 

The Florida case clearly shows the loopholes in the current operating system. But with an AlarmAgent cloud-based monitoring and alarm system installed on-site,  the water quality and safety can be monitored and alerted when drinking water is impacted. The best practice is to have active human monitoring supplemented with the right system of quality monitoring and alarms to notify those operators if they miss something critical. After all, we are all human and can make mistakes.

#3. Monitoring and Record-Keeping

Many water system treatments do not have the necessary infrastructure to check all the parameters to identify threats at their most preventable or reversible stage.

Many facilities rarely record the data correctly, making it challenging to audit what happened after the fact, limiting the ability to learn from situations that occur, and putting a plan to prevent future attacks.

Generally, we assume that having a feed, an excellent membrane unit, and a pre-treatment design is enough to keep the system running independently.

The truth is, these systems should not be left alone on their own unless a cloud-based alarm system is fully installed on the premise.

It is always important to monitor specific parameters such as:

• UV intensity
• Normalize Permeate Flow (NPF)
• Silt Density Index (SDI)
• The Pressure Across Stage

Without recording and monitoring, the chances that system performance will drop are very high.

#4. Maintenance

Lack of maintenance can be detrimental to a water treatment facility. Sadly, many water treatment facilities in the US are underfunded, meaning there is not enough to carry out regular maintenance on equipment. The budgetary impact of COVID-19 is still unclear in many jurisdictions, but it is safe to assume that already tight budgets will come under even more pressure. 

It is essential to initiate a continual maintenance process in the water purification systems to keep operating expenses low over the long-term. For instance, changing the pre-filtration cartridges and bags prolong the life of the membranes within the system.

#5. Lack Of Qualified Personal

Water facilities can correct most of our treatment problems with the experts’ help, but those experts are in short supply.

Water treatment is a complex process. Without training, teams are at risk of misunderstanding the process.

Many of these facilities are manned by unskilled or semi-skilled personnel, which itself is not a problem. Lack of funding for adequate training and mentorship is the problem.

Solutions To The US Water System Issues

The Florida water system hacks are just the most recent attack on our water systems. Situations like this will only become more commonplace. Instead of waiting for another attack, operators must prevent future attacks by improving these systems' security.

Here are some of the things that need to be done.

• Increase the number of trained personnel in water plants.
• Ensure that every public facility has adequate staff on the ground.
• Encourage monitoring of system parameters.
• Restrict remote access to systems only where necessary.
• Encourage the use of secured software in both large and small facilities.
• Upgrade smaller facilities to a level that makes future attacks difficult.
• Upgrade equipment to modern designs capable of self-regulation.
• Supplement the human monitoring that must occur with an industry-trusted tracking and alarm system, such as AlarmAgent.

Above all, you need to install the RACO alarm system for water treatment systems for quality and performance.

RACO Alarm System For Water Treatment Systems

RACO Manufacturing & Engineering, Co. makes the AlarmAgent wireless remote telemetry unit (WRTU) which, when installed, will monitor water quality within any water facility and the water distribution system, large or small. 

How you design and install an AlarmAgent system can make a big difference. Ideally, monitoring can occur both within the facility as well as out in the field. It is essential that your alarm monitoring system is close to the residents who receive your water. This allows you to ensure that both intentional and accidental contamination can be caught before people are affected. It ensures that no one consumes contaminated water.

Additionally, the AlarmAgent system is intentionally sensor-agnostic and can be set up with any existing technology you may be using. Raco only cares that you can monitor water quality and alert you anytime there is danger.

7 Reasons To Choose RACO Alarm System for Water Treatment Systems

AlarmAgent is the best in the market. It is recommended for performance and quality. There is a long list of happy municipal and industrial customers who will vouch for its value.

For a reasonable price that can fit into even the tightest budget, AlarmAgent can put water and wastewater operators in a position to monitor the public water system's welfare and ensure customers’ safety.

There are (at least) seven reasons why you should use AlarmAgent for water and wastewater treatment systems, and they include:

#1. Real-Time Reporting

AlarmAgent collects real-time data  24 hours a day, at intervals that can be configured as frequently as every five minutes. You'll have access to RTU health reports, M2M outputs in easy-to-understand HMI reports, and user-friendly dashboards to keep everyone at the facility on the same page. 

#2. Wireless RTU 

AlarmAgent is a wireless RTU that is easy to install. They are more proficient than landlines, and our units have great coverage across North America. If a land-line solution is what you need, Raco also offers the Verbatim, a trusted line of alarm auto-dialers that will fit your needs as well.

#3. Top-notch Monitoring and Control Functionality

An upgrade for less than a dollar a day can provide cloud-based SCADA functionality with the flip of a switch instead of a costly SCADA implementation project. The system can receive the status of all channels, including analog, in seconds. With RACO, you can initiate a simple configuration and upgrade in seconds.

It features an operating template, eight digital and two universal inputs, and two relay outputs.

#4. Intelligent System Indicator

It gives immediate status updates and alarms for several actions such as transmission, local service registration, signal strength, armed/disarmed, etc. Administrators can easily configure what scenarios they want to escalate to alarms and who needs to be contacted, and how. 

With RACO, your system is never caught unaware. Our alarm system provides you with information about potential threats to within and outside the water facility.

#5. Flexible Alarm Notification & M2M Control 

With instant alarm notification by voice, SMS, pager, and email, operators can control locally connected devices with simple relays if they want to take action on alarms automatically.

RACO makes system updates easy for you to see. 

#6. 24/7 Peace of Mind

AlarmAgent is one of the most efficient ways to step up your facility security. It allows you to retrieve equipment and status info anytime you want to.

In addition to this, Raco customer support gets excellent grades for its service and responsiveness. 

#7. Access to AlarmAgent.info

AlarmAgent features a complete SaaS application that makes AlarmAgent.com OPC-compatible. This means you can install your system without any difficulty.

In addition to this, you can share data with their HMI / SCADA software and third-party applications. 

Conclusion

The intrusion on the water facility in Florida should put everyone in the water wastewater industry on notice. We are high-value targets for bad actors. Having systems to detect, prevent, and deter these attacks are critical. Having backups and other systems to monitor and notify people when something bad is happening can save lots of time and money, but more importantly it can save lives. 

Contact a Raco sales rep to discuss your needs as a system operator and how the Raco family of products can put you in a position to avoid the next big headline.