September 8, 2017

Capacity Planning for Next Generation Utility Networks (Part 1)

An analysis of utility applications, capacity drivers and demands

Utility networks are going through massive transformations towards next generation conduits for new applications and critical data. Many of these networks have been around for years and have adequately handled traffic demands, but as new services arise so does the demand for capacity. Networks that were built to sustain 100-200Mb/s of TDM/SONET/SDH traffic are now facing bottleneck situations and increased pressure to grow. The migration to packet based networks has introduced a plethora of new challenges to Utilities regarding capacity planning, efficiency, reliability, and cost.

This paper describes capacity drivers for next generation utility networks and delves into detail about how to design these networks to meet the capacity need without sacrificing reliability or increasing cost. In the first part of this white paper, we will examine each of these services and its capacity contributions and then in the second part, explore how the combination of these services affect the network backhaul design.  Separately we will do an analysis of a typical network and how to dimension the network capacity to be able to adequately handle these services and some factor of future growth.

Utility Capacity Drivers

Traditionally, private utility networks have been used to carry mission critical applications which are too sensitive to be carried on public operator infrastructure. These include mostly voice, SCADA, teleprotection, operational services, and security related traffic.  All of these services have been relatively low capacity in nature and even for larger networks with many substation and relay sites, the aggregation of all of these services didn’t yield a need for a high capacity backhaul pipe (larger than a standard DS3).  Over the years, some services have grown in terms of scale and are requiring more bandwidth.  New services have also been introduced into the network and are redefining how much capacity is needed. Some of the key applications and services, old and new, that are present in network planning and causing a huge spike in the capacity planning phase are:

1. Teleprotection (where low latency = increased bandwidth required)
2. SCADA (migrating to packet-based protocols such as DNPoIP)
3. Traditional and Emergency Voice (moving to VoIP solutions)
4. Operational Services (remote access to an increasing number of field devices with increased frequency)
5. Corporate Networking (field staff expectation to be connected without interruption)
6. Security (both physical security systems and video surveillance backhaul)
7. Phasor Measurement Unit (PMU) Services (highly-accurate, real time sampled data – next generation SCADA)
8. LTE data (future)

Teleprotection:
Teleprotection is one of the most critical services for utilities and while each is inherently low in capacity (one DS0 or less), the method of transport of teleprotection can impact the necessary bandwidth.  These circuits prevent failure and severe damage in the power network by transferring command signals between substations in the event of a protection trip.  The circuits are extremely time sensitive to ensure that the faulty part is disconnected within the shortest possible time.

In terms of transport, traditional networks have carried teleprotection over TDM networks, where DS0’s were natively transported and service performance was guaranteed.  As networks are migrating to packet based systems, the transport of these circuits over packet infrastructure can introduce issues with latency and delay.  This occurs when the native signal is encapsulated and carried over packets in a process known as Pseudowire Emulation (PWE).  PWE requires additional processing with adds latency to the transmission of the the signals and one of the ways to counter the latency issue is to carry the signals over smaller frame size packets.  While this can reduce the latency of the signal, it introduces additional overhead traffic to the overall payload.  Since these circuits are small, the individual addition of unusable overhead traffic is insignificant but from the total network standpoint, it can be noticeable.  The capacity requirements per substation can vary significantly depending on the number of Teleprotection channels required.  Smaller substations might only require a handful of circuits while busy and high voltage substations could have as many as 75 channels terminating at the site.

Supervisory Control and Data Acquisition (SCADA):
SCADA traffic represents the command and control network of the utility and like Teleprotection is inherently very low in capacity but high in importance.  SCADA is used to both monitor and control how the power system operates and provides operating status and alarms when abnormal conditions exist.  This traffic provides data on the characteristics of the network that allow the utility to operate more reliably while also lower operating costs because of real time problem detection and resolution.  It is the real-time dashboard of the grid, and hence is mission-critical.

Traditional and Emergency Voice:
Utilities still maintain private networks for their internal and mission-critical voice communications including traditional push-to-talk (PTT) mobile radio, private branch exchange (PBX) lines, and newer voice systems like VoIP and LTE. Hardened, private infrastructure ensures mission critical voice is available in emergency situations.  Sizing the network for voice applications requires understanding current requirements and ensuring there is adequate space for future services and users.  In terms of capacity, networks are designed for peak load since while the usage of voice is intermittent and not as frequent as in commercial LTE networks, the requirement can increase in cases of emergencies or if there is aggregation at the site.

Operational Services:
Operational services include data access applications like metering, fault reporting and event analysis, which are all individual low capacity services but once aggregated, can become a large consumer of available bandwidth.  This is especially true of metering services such that if the requirements change to include more meters in the network or more frequent polling of meters are desired, the increase in bandwidth also increases exponentially. While many utilities offload metering traffic to public networks, for security reasons, more and more are looking to keep this traffic on their own networks.

Corporate Networking:
With the availability of Ethernet pipes on a private network, applications and services that sustain the corporate environment will increase. This includes standard LAN services, Internet connectivity and connectivity to link substations together.  Traditionally this traffic has resided on public networks however access to a high speed private network offers a lower cost and more reliable option and many utility networks have a growing volume of corporate traffic.  The bandwidth of this type of traffic can vary depending on the type of applications and these services are often times the lowest in priority but are still very capacity hungry.

Security:
Security might be the single largest driver of capacity in utility networks today and the most bandwidth intensive security application is surveillance video.  With the need to protect critical infrastructure and meet new regulations, use of surveillance cameras has risen. Networks need to support multiple, often high definition, cameras per location- each capable of 1080p resolution and also refresh and operate at a rate of 60+ fps.  Newer cameras also offer pan-tilt-zoom (PTZ) capabilities allowing complete remote control of the camera. Each of these types of cameras can require up to 6Mb/s of sustained bandwidth. Utilities with thousands of substations, generation sites, and other facilities will demand ultra high capacity networks and will need to be carefully planned and engineered to ensure capacity demands of worst case emergency scenarios are met.

Phasor Measurement Unit (PMU) Services:
PMU sensors give utilities additional details about the performance of the network and track power quality at a very high resolution by streaming real time data as opposed to collection and transmission over time intervals.  PMU’s are considered the next generation of SCADA because of its increased level of granularity and details and will eventually become one of the most important data sources in next generation power networks. Comparing the data resolution of PMU’s to SCADA, while SCADA typically collects 1 sample every 2-4 seconds, PMU’s collect 10-60 samples per second and measure both magnitude and phase angles of a phasor signal (SCADA only measures magnitude). The adoption of PMU’s in a network is growing and they will initially coexist with SCADA in the network but eventually PMU’s will remain.  Designing a PMU network can also affect the bandwidth the services require and in terms of PMU design, there are two architectures – central concentration and local concentration.  In a central concentration architecture, PMU’s forward data to a main control site while in a local concentration architecture, PMU’s provide primary data to the local substation and then a separate signal is sent from that substation to the control site.  The following table shows an extrapolation of traffic depending on the architecture for local versus central concentration.

Figure 1 – PMU bandwidth requirement for central office.¹

The bandwidth estimates show the approximate bandwidth required at the control site for each of the architectures.  Also based on the size of the network, the number of PMU’s in a fully-deployed network can easily exceed 500.

LTE Data:
Just like public safety, utility teams would benefit from mobile broadband services. Currently, utilities lack dedicated spectrum for such services. Despite this, some utilities are forging alliances and partnerships to gain access to LTE spectrum for broadband communications. In these cases, network wide bandwidth can be difficult to plan since capacity requirements will depend on site density, spectrum availability, and LTE network design. Regardless, mobile data can drastically increase bandwidth requirements on utility transport networks.

Summary

Utilities require ultra-high capacity networks to deal with growing volumes of mission critical data.  While “backbone” links in older networks were designed with 50Mb/s, newer networks can easily require a minimum of 200Mb/s growing to upwards of 1Gb/s. Factors like the size of the network and the volume of services will affect these estimates.

If you have comments or input on any of these figures, we’d love to hear from you. Please comment on the blog below or email us at marketing@aviatnet.com.

Stay tuned for part 2 in this blog series where we offer some detailed planning guidelines and capacity dimensioning for utility transport networks.

¹Schweitzer Engineering Laboratories, https://selinc.com/solutions/synchrophasors/report/115281/.