Cybersecurity in the Power Grid 
–A 360° View | Part 2

Welcome to the 2nd episode of our Energy Talks miniseries, called Cybersecurity in the Power Grid, in which we provide you with a 360-degree view of how power grids can best safeguard their infrastructures from cyber attacks.

In Part 2 of this miniseries, we engage with OMICRON application engineers Ozan Dayanc and Thomas Wolf to explore the multifaceted world of power network security.

Discover the challenges faced by power providers and practical recommendations for enhancing OT network security. From the intricacies of daily work to customer interactions, Ozan and Thomas share insights into power network cybersecurity.

Stay tuned for upcoming episodes in our Cybersecurity in the Power Grid miniseries.

Listen to the podcast episode

Scott Williams Welcome to Energy Talks, a regular podcast series with expert discussions on topics related to power system testing, data management, and cybersecurity in the power industry. My name is Scott Williams from the podcast team at OMICRON and I will be your host. Hello, everyone. Welcome to our special Energy Talks miniseries called Cybersecurity in the Power Grid, in which we provide you with a 360 degree view of how power grids can best safeguard their infrastructures from cyber attacks. 

In our second episode of this miniseries, OMICRON cybersecurity experts Ozan Dayanc and Thomas Wolf describe the critical role that intrusion detection systems (IDS) play in Operational technology (OT) networks, as well as the importance of adopting holistic cybersecurity solutions for securing power networks based on their personal journeys and other side findings. Ozan and Thomas highlight the challenges faced by power providers and offer practical recommendations that encompass not only software solutions, but also the range of specialized services OMICRON has to offer to power providers for cybersecurity. 

Hello, Ozan. Hello, Thomas. Welcome to Energy Talks. 

Ozan Dayanc Thank you for having me. 

Thomas Wolf Thank you for the invitation. Scott. 

Scott Williams Thank you again for joining me. Could you both introduce yourselves and share the experiences that led you to become application engineers at OMICRON? What sparked your interest in power network security specifically, and how has it shaped a professional journey? 

Ozan Dayanc Sure, I can start. 

Scott Williams Okay. Thank you, Ozan. 

Ozan Dayanc Absolutely, as you mentioned. My name is Ozan Dayanc, and I joined the OMICRON team in 2020, stepping into the role of a cybersecurity application engineer. My journey into the realm of electrical engineering began during my time at a technical high school, and this path continued through my university studies. Upon completing my bachelor's degree, I found myself deeply involved in a substantial project focused on railway systems. In this capacity, I shouldered responsibilities ranging from configuring network switches to managing servers like DNS Active Directory, all while implementing cybersecurity measures to fortify the system. It was during this pivotal project that I started grasping the essence of cybersecurity and its broad-reaching implications across diverse network components.

Upon the conclusion of the Railway IT project, I was tasked with a new mission: commissioning a substation automation system. This endeavor brought me back to my core field, albeit with a newfound emphasis on security. The insights gleaned from my previous IT project proved invaluable in meeting the cybersecurity demands of these fresh OT projects. I dedicated myself to the commissioning of IEC 61850 and IEC 6870-5-104 systems, spanning from SCADA to relay. Additionally, I took charge of configuring network switches and enacting stringent security protocols. Following two years immersed in this dynamic field, predominantly in Egypt with occasional assignments in Germany, I arrived at a pivotal juncture. I yearned to amalgamate my knowledge in both IT and OT, embarking on a journey of continuous learning while concurrently contributing my expertise. This aspiration guided me to the doorstep of OMICRON, where I find myself today.

Scott Williams Well, very good. Thank you. Ozan. Thomas, what about you? 

Thomas Wolf Certainly, Scott. I'm Thomas Wolf, and I've been a part of the OMICRON team for a little over two and a half years now. In my role as a sales and application specialist, my primary responsibility revolves around the cybersecurity solution StationGuard. Prior to joining OMICRON, I amassed nearly 15 years of experience working as an engineer specializing in substation automation systems across various companies.

My journey commenced in 2005 at Aveva, and through several progressive steps, I ventured into roles with prominent organizations such as Schneider Electric, Alstom, and NGE. During my tenure, I actively contributed to numerous projects, assuming the mantle of a technical project leader. This role placed me at the helm of engineering and commissioning activities related to substation automation systems.

Drawing from this extensive background, I can confidently assert that I possess a comprehensive understanding of the intricacies surrounding substation automation systems. This insight spans the operational dimension, the engineering facets, and equally importantly, the cybersecurity landscape. It's this amalgamation of perspectives that fuels my motivation to incorporate this wealth of knowledge into our product development and cybersecurity initiatives.

Scott Williams Very good, Thomas, Thank you for that. Okay. So, could you both walk us through a typical workday as an application engineer with a need for frequent travel? What are the primary objectives you aim to achieve while onsite? 

Ozan Dayanc I think I can start again. My workdays come in two variations: those I spend in the office and those I'm on-site with the customer. When I'm in the office, I begin by reviewing emails to check if any customers require support from our team. I also maintain a close collaboration with our development team, actively participating in their daily meetings to stay updated on the progress of our product. This approach not only allows me to develop a deeper understanding of the product but also provides valuable user feedback to the developers that can influence future development.

Another significant aspect of my role is conducting security assessments for our customers. Our work primarily involves networks in control centers, substations, and power plants for energy suppliers. Based on customer requests, we install an intrusion detection system called StationGuard into their network, and we periodically perform security assessments. These assessments involve analyzing network setups, assessing system hardening, reviewing segmentation, authentication, definable abilities, and creating an asset inventory. From this analysis, we create a comprehensive report summarizing the network's security status, which we then present to the customers. This analysis is conducted in collaboration with our cybersecurity analysts.

So, this is more or less what I do in the office. When I'm on-site with a customer, whether it's a substation control center or a power station, my tasks typically include configuring our IDS system and integrating it into their network. I also provide training to the users, which involves network administrators who are familiar with the network to verify the configuration. IDS systems don't operate effectively without proper configuration. Furthermore, they rely heavily on network equipment like switches, which must be correctly configured since IDS systems depend on span ports, also known as mirror ports. Another challenge is that we cannot make changes on the customer's site. These are highly restrictive environments. Therefore, ensuring the presence of a responsible network engineer like a system admin who can assist with switch configurations is essential before on-site visits. These are just a few of the challenges we encounter. However, being on-site is generally enjoyable, especially given my previous job, which also involved frequent on-site visits.

Scott Williams Thank you, Ozan. Thomas, could you walk us through your typical workday? What are your primary objectives when you are on site at a customer location?

Thomas Wolf Describing the workday couldn't be more accurate than what Ozan mentioned, as we have more or less the same tasks whether we're on-site or in the office. What I can emphasize is that nearly every week, if not every day, brings something different. This variety comes from working with various customers on different projects, each presenting its unique set of challenges. Our aim isn't just to convince customers of our cybersecurity solution; we also strive to efficiently install and configure the system, delivering the best results for the customer.

These minor challenges in each project provide us with valuable learning opportunities and help us expand our knowledge of substation automation systems and cybersecurity. However, our ultimate goal, regardless of the challenges, is to ensure that our IDS system functions effectively and successfully in the customer's network.

Scott Williams Thomas, Thank you for that. So, both of you, when customers approach you or when you proactively reach out to customers, what are the key motivations behind these interactions? Are regulatory compliance and system malfunctions the primary drivers or are there other factors involved? 

Thomas Wolf In general, my primary motivation is to look after our IDS solution and implement it along with the necessary network infrastructure to protect and monitor the network, ensuring the safe operation of both the system and the network. However, in the German market, the predominant motivation for customers to approach us is regulatory compliance. They require a cybersecurity solution, particularly our intrusion detection system, StationGuard. This heightened demand is a result of the German law, known as the IT Security Act 2.0 or IT-Sicherheitsgesetz in German, which came into effect on May 1st, 2023.

According to this law, every provider of critical infrastructure, such as substation automation systems, control centers, or power plants, is mandated to install an intrusion detection system within their network. Furthermore, the legislature has clearly defined the responsibility on the management side and increased the penalties for non-compliance. Consequently, some customers are feeling the pressure to evaluate IDS systems and promptly implement them in their networks.

In German-speaking countries like Switzerland and Austria, where energy transmission providers also connect to the German energy grid, customers are initiating the evaluation process to find an IDS solution and integrate subsystems into power plants to ensure compliance with German law. We've already successfully deployed our IDS solution in some substations for customers in Switzerland and Austria as well.

Scott Williams Very good Ozan. Thank you. Okay, drawing from your on-site experiences, what are some noteworthy findings and challenges you have encountered with power networks? Could you share any personal learnings from these experiences? 

Thomas Wolf Certainly. Larger and more complex networks tend to bring forth bigger challenges. Moreover, age often plays a significant role in the complexity of issues we encounter. Our process usually begins with the installation phase, and during this phase, we dive into the project's documentation. However, some findings can be unearthed even before the installation, often during the "proof of concept" stage. As we prepare for installation, we meticulously review the system's documents, and this careful examination frequently reveals gaps in information or documentation errors.

Common issues we identify during this phase include missing or improperly formed SCD files for 61850 substations, outdated IP address lists, and discrepancies between the system diagram and the actual installed system. Communication lines may be missing or incorrect, and the documentation related to Intelligent Electronic Devices (IEDs) can be inadequate, particularly when it comes to their intercommunication. Another challenge arises when customers want to designate a mirror port on a switch for the IDS system but find that no Ethernet port is available due to all ports being in use, a result of insufficient information in the documents.

It is paramount for us to have all documents, especially updated ones, readily available when we embark on configuring and installing the system. As we proceed with the installation and commissioning of the IDS, we encounter typical findings. These findings encompass unknown IEDs, network adapters that don't align with the configuration file inside the device (especially in the context of IEC 61850 substations), and issues related to IEC 104 SCADA communication, such as encountering unknown data types or discovering unused or unnecessary operations like set point operations.

In some instances, we also detect unused protocols within the network, such as NetBIOS or license applications. Addressing most of these problems is more viable during the commissioning phase of the systems. Fixing these issues or making changes to device settings becomes significantly more challenging for customers once systems are in operation. For instance, in many substations, there are devices with outdated firmware, and customers might not be able to update these devices easily. Attempting updates might necessitate updating all devices in the substation, which is not only time-consuming but also costly.

Hence, we offer support for the engineering and commissioning phases of substation automation systems, providing services during commissioning and offering troubleshooting assistance during this critical phase. Often, customers retrospectively express that conducting the IDS installation or proof of concept during the commissioning phase of the substation could have mitigated many issues faced during the operational period.

Ozan Dayanc Yeah, I agree with Thomas, and I'd like to reiterate some of the key points he made, as my experience aligns with his perspective. However, I'd like to add a brief note regarding the complexity that arises as networks age. In the realm of OT, you often encounter networks that have been in operation for decades, some stretching back 20 years or more, with minimal changes or updates over time. One recurring issue we confront in such scenarios is a significant lack of documentation. This includes missing IP address lists, outdated network diagrams, and various other documentation gaps.

Another challenge stems from the shifting landscape of system knowledge. In many cases, the original system engineers who commissioned these aging networks have moved on—often due to retirement or other career changes. Consequently, critical knowledge regarding the network's communication framework, device functions, and overall network design can be lost. These details are essential when configuring an Intrusion Detection System (IDS) within the network.

In these older networks, we frequently encounter numerous Remote Procedure Call (RPC) problems, characterized by extended recalculation times. This results from a lack of ongoing maintenance, improvements, and troubleshooting efforts over the years. Unlike Information Technology (IT) networks, OT networks can remain unchanged for extended periods, making them significantly distinct in terms of network dynamics.

Scott Williams In terms of power network standards, how do German speaking countries differ from other countries in Europe? Are there notable variations, such as contrasting approaches to the protocols they are using or stricter government regulations that our providers should be aware of? Thomas, can I ask you to start? 

Thomas Wolf Of course, thank you. Transmission and distribution grid providers in German-speaking countries possess a vast installed base, shaped by a long and storied history. Over the decades, they've implemented various substation automation systems with differing philosophies of signal exchange. Initially, hardwired signal exchange methods prevailed, evolving into aerial communication, which gained prominence in the 1980s. This era saw the emergence of communication protocols like IEC 103, driven by protection devices. Presently, Ethernet-based communication, notably IEC 61850, has taken precedence.

This coexistence of diverse communication philosophies has created a complex landscape. Many substations in German-speaking regions still rely on older communication standards like IEC 101 and IEC 103 for protection and SCADA communication. For some customers, the transition to Ethernet-based communication using IEC 61850 is an ongoing process. Transmission grid providers, in particular, maintain a high level of standardization across all substation components, including protection IEDs, cubicles, wiring, and communication protocols.

This commitment to rigorous standards, coupled with the extensive history of the installed base, presents challenges when attempting to swiftly shift communication philosophies from aerial to Ethernet-based. Consequently, the transformation and refurbishment processes are more time-consuming than in other regions, impacting the evaluation of an IDS as well.

Scott Williams Very good. Thank you, Thomas. Ozan?

Ozan Dayanc Certainly, I can provide insights into this topic with a focus on two key aspects: regulatory standards and protocol standards, considering the business activities across different EU and non-EU countries.

As Thomas mentioned earlier, Germany has witnessed significant regulatory developments that power providers need to be mindful of. The introduction of the IT Security Act 2.0 has particularly impacted critical infrastructures. Additionally, there is a new EU-wide directive known as NIS2, which aims to enforce cybersecurity regulations across the European Union, with a specific focus on critical infrastructure. Member states are expected to regulate this directive by October 17, 2024. These regulatory changes are likely to lead to increased investments in cybersecurity.

Moreover, there are internationally recognized industrial control system security standards, such as IEC 62443, which cover a wide range of cybersecurity aspects, including technical measures and security requirements for suppliers, among others. These standards play a pivotal role in ensuring the security of critical infrastructures.

Turning to the second aspect, which relates to protocol standards in OT networks, IEC 61850 standards stand out, particularly at the substation level. This protocol is considered one of the most advanced in public grid networks and is widely adopted by utilities worldwide due to its numerous advantages. While communication between control centers and substations still involves several protocols like IEC 101 and its DCP IP version IEC 104, it's important to note that countries like the UK, the US, Australia, and Canada often rely on the DNP3 protocol for control center communication, distinguishing them from other regions.

In summary, although there are variations in regulations, standards, and protocols from region to region, the fundamental principle of operating and protecting a power grid remains the same. Power providers must stay informed about these variations to ensure compliance and effective operation within each specific region.

Scott Williams Thank you, Ozan. Considering the significance of cybersecurity solutions, what recommendations would you give power providers to enhance the security of their OT networks? Ozan, could you start with this, please? 

Ozan Dayanc Absolutely, I'd like to emphasize some organizational solutions instead of merely focusing on technical problems. One of the key issues we frequently encounter pertains to the scarcity of skilled staff and resources within the realm of OT security. Often, within an OT team, there's understandable reluctance to take on additional responsibilities because team members are already stretched thin. I believe Thomas would concur with me on this issue. While this situation is quite comprehensible, it underscores the critical need for dedicated cybersecurity roles. This challenge is pervasive in many organizations.

Furthermore, the lack of skilled personnel can also be attributed to the failure to present a clear cybersecurity strategy to the board and subsequently recruit new employees. This stands as another primary obstacle. Additionally, a notable divide often exists between IT and OT departments. These two departments, along with their staff, may be unaware of each other's activities. Sometimes, IT may seek to incorporate OT networks into their cybersecurity program without notifying the OT department, and vice versa. OT might initiate security projects without seeking input from the IT department. Ideally, both departments and their team members should harness each other's expertise, with the overarching goal of safeguarding OT networks. Collaboration and knowledge sharing are paramount for success in this regard.

Scott Williams Ozan, thank you. Thomas, what recommendations do you have for power providers to enhance the security of their OT networks? 

Thomas Wolf Yes, I fully concur with Ozan and his points. Indeed, bridging the gap between IT and OT is one of the challenges we face. When it comes to installations and working within substations, we have to collaborate closely with both teams. As Ozan rightly mentioned, there are sometimes disparities in the knowledge held by IT and OT professionals, and it falls on us to bring them together.

Scott Williams Ozan and Thomas, building on your previous responses, it's evident that the divide between IT and OT is a known issue. While OT has begun to implement cybersecurity practices, there still appears to be a conflict regarding cybersecurity responsibilities within OT. In light of this, who should assume the responsibility for cybersecurity in the OT segment of organizations?

Ozan Dayanc As I mentioned earlier, addressing the cybersecurity gap in OT necessitates the allocation of appropriate resources and the establishment of a dedicated OT security team, comprising both IT and OT professionals. Ideally, this team would include a cybersecurity officer working closely with SCADA or protection engineers—a sort of dream team. The primary objective would be to protect the OT domain and act as a bridge between these two departments. I've witnessed successful implementations of such teams in various organizations, particularly in the power grid sector. It's essential to grasp that the burden of cybersecurity should not be placed solely on either OT or IT individuals. Effective cybersecurity demands a collaborative approach, leveraging the expertise of both groups to formulate a robust defense strategy.

Scott Williams Alright, and Thomas? 

Thomas Wolf Well, I can draw a parallel to how protection engineers and SCADA engineers used to be distinct roles in the past. Similarly, we often see a separation between OT and IT engineers. In the past, a protection engineer's focus was primarily on protection functions, configuring settings within the device, and their responsibility typically extended to the inputs and outputs of the IED. On the other hand, SCADA engineers were primarily concerned with communication protocols and signal exchanges between components, such as IEDs, RTUs, and HMIs.

However, as Ethernet-based communication became the standard for substation automation systems and SCADA systems, particularly with protocols like IEC 61850 and IEC 104, protection engineers and SCADA engineers began working more closely together than before. This parallels the convergence of OT and IT engineers when it comes to cybersecurity. There are many overlapping issues related to communication, time synchronization, and the critical nature of communication and signals within the system. It's crucial for everyone to understand what's happening in the other part of the system.

For example, IT personnel should grasp the significance of a missing GOOSE (Generic Object-Oriented Substation Event) signal. It's not merely a protocol issue; it's not just about a web browser not being accessible. Understanding that a missing GOOSE signal can pose a significant risk in a substation, such as the failure of a critical protection trip signal, is essential.

Conversely, OT professionals should recognize the criticality of using secure communication and protocols within the substation environment. This includes avoiding insecure protocols like Telnet or HTTP, which can introduce vulnerabilities.

In general, when it comes to cybersecurity, there should be no rigid boundaries between OT and IT. The cybersecurity approach functions optimally when these teams collaborate closely.

Scott Williams Thank you for that perspective. Now, looking ahead, what do you foresee as developments in the future of power network security? Are there emerging challenges or opportunities that power providers should prepare for? Ozan, what's your take on this?

Ozan Dayanc I anticipate that in the future, there will be a greater convergence of IT technologies, products, and protocols into OT networks. This trend is already visible in advanced substation automation systems networks, where components like Active Directory servers, virtualized appliances, IDS implementations, firewalls, database servers, asset management tools, data aggregators, and more are becoming integral parts.

One area of development is the growing importance of IDS in monitoring and securing IT networks. IDS not only helps in detecting intrusions but also offers several other advantages, such as asset inventory management, vulnerability management, malfunction identification, and functional monitoring. These capabilities are particularly valuable for OT professionals.

All these trends indicate an ongoing evolution of OT networks toward digitalization. This trend is visible in new or refurbished substation automation systems across various countries. OMICRON closely monitors these market changes and actively contributes to this transition by providing solutions and services. Our solution offers visibility into OT networks, assisting with asset inventory management, vulnerability management, and intrusion detection.

Additionally, our engineering service team supports utilities during the initial phases of substation automation system projects. We help with tasks such as specifying requirements, selecting appropriate technologies, and designing secure-by-design networks.

Our overall strategy aims to ensure that power grids are well-prepared to tackle the challenges and seize the opportunities in the evolving landscape of power network security.

Scott Williams Very good. Okay, Thomas, what is your opinion? 

Thomas Wolf For new substations or the refurbishment of existing systems, cybersecurity has become an integral part of the project. Right from the planning and design phase, a dedicated team, comprising mostly IT and OT experts, is involved. These experts serve as our direct contacts in the project and play a pivotal role in defining the cybersecurity requirements for the entire system.

These requirements are then incorporated into the technical specifications of the tender documents. The focus of these experts extends to optimizing and simplifying network architectures from a cybersecurity perspective. This includes aspects like network segmentation, redundancy, firewall concepts, and more.

Additionally, there's a critical emphasis on how to efficiently implement Intrusion Detection Systems (IDS) into the system. Ensuring the overall system's availability is another vital aspect that remains a top priority during the design and configuration phases.

Looking ahead to the future, especially in projects involving full digital substations with IEC 61850 process buses or the use of merchant units in Sampled Values, where the entire communication relies on Ethernet, the functional aspects of availability, complexity, efficiency, and cybersecurity requirements present even greater challenges than before. However, OMICRON has a team of experts prepared to provide services and support to customers in this evolving landscape.

Scott Williams Very good. How do OMICRON's OT cybersecurity solutions differ from other companies in the monitoring business?

Ozan Dayanc Right from the inception, our focus has been on bridging the gap between IT and OT domains. At OMICRON, we possess extensive expertise in both areas, and we've harnessed this knowledge to develop our IDS solution, StationGuard. What sets StationGuard apart is its universal appeal to both IT and OT users, thanks to our deep-seated proficiency in both domains.

Furthermore, StationGuard boasts unique advantages tailored to OT professionals, incorporating inherent OT knowledge. This capability plays a pivotal role in reducing false positive alarms, a persistent issue in the IDS landscape. For more details, Thomas can elaborate further.

It's crucial to recognize that our commitment goes beyond just offering solutions. We provide comprehensive after-sales services, which include the analysis of alarms and events generated by StationGuard. While we aren't a security operations center provider, we collaborate with our customers to assess their events. Our multidisciplinary team, equipped with both IT and OT knowledge, is essential for precise data analysis and comprehending outcomes. Effective communication is the linchpin, and our ability to speak our customer's technical language enhances the value of our feedback process.

In summary, our solution is meticulously crafted for OT networks, affording us a distinct advantage in effectively monitoring OT networks. By amalgamating our IT and OT expertise, we deliver comprehensive and specialized solutions tailored to address the unique challenges of the OT environment.

Scott Williams Thank you Ozan. Thomas, would you like to add anything? 

Thomas Wolf Yes, certainly. At OMICRON, we have extensive experience and profound knowledge regarding communication and the functional processes within substations, control centers, and power plants. This expertise served as the foundation for developing an IDS system tailored to the needs of power utilities, known as StationGuard. Consequently, we assert that our IDS system, StationGuard, is purpose-built for use in typical communication environments, precisely those encountered in substations, control centers, and power plants. Unlike most of our competitors, who predominantly hail from the IT realm, we approach IDS from a unique perspective.

Our key differentiation lies in the ability to minimize IDS installation time significantly. In general, it takes just several minutes to a few hours, meaning we can swiftly provide solutions, whether for new systems or those already in operation. Furthermore, customers have full control over the installation and configuration of the IDS. They can opt for our support or carry out changes themselves as needed, even without strict onsite assistance."

Scott Williams Ozan and Thomas, before we conclude, may I ask if you have any final thoughts on why OT networks require an IDS? 

Thomas Wolf Of course. Cybersecurity isn't merely a product; it's a philosophy within a project. It's an ongoing journey that persists throughout the entire project's lifecycle. At OMICRON, our aim is to accompany our customers on this path.

Scott Williams Very good. Ozan?

Ozan Dayanc Yes, as Thomas mentioned, cybersecurity is an endless journey that repeats itself. Utilizing IDS for monitoring offers numerous advantages in terms of cybersecurity and functional monitoring. What's even more important is employing an IDS with built-in OT knowledge, which provides unique benefits not only for IT but also for OT professionals. OMICRON Electronics is the right destination to explore these solutions.

Scott Williams Ozan and Thomas, thank you both for joining me in this second episode of our Energy Talks miniseries on cybersecurity.

Thomas Wolf Thank you very much. 

Ozan Dayanc Thank you very much. 

Scott Williams A big thank you to our audience for tuning in to this episode of Energy Talks. We always welcome your questions and feedback. Feel free to send us an email at podcast@omicronenergy.com. With several years of experience in power system testing, data management, and cybersecurity in the power industry, OMICRON offers tailored solutions for your applications. For more information, please visit our website at www.omicronenergy.com. Join us for the next episode of Energy Talks. Goodbye for now, everyone."