Reliability standard ain’t broke

Since market start the NEM has used a single and simple customer outcome metric for its reliability standard: “Unserved Energy” (USE). This means the share of customer demand that cannot be supplied due to a reliability event, i.e. AEMO directed rotational load shedding. Reliability excludes the larger causes of customer interruptions that arise in local networks or transmission grid security events, such as the Queensland 25 May 2021 event.

The reliability standard is 0.002% of USE in a region over a financial year, or, in other words, we target an average of no more than 10.5 minutes off supply per customer per year from rotational load-shedding. This is about one tenth of what distribution networks target for their customers. A more conservative, “interim reliability measure” of 0.0006% USE also applies to some market mechanisms.

With respect to assessing performance, a USE total is as simple as a measure can get. And by avoiding arcane power system concepts like “N-1” or “loss of load probability”, it goes directly to what the power system should be focussed on, customers, and is easier to explain to lay stakeholders.

With respect to forecasting reliability, AEMO simulates future power system years with probabilities of adverse demand and supply events. Some simulations will include periods of shortfall that, if they were real, would cause rotational load shedding. If the average amount of USE across all the simulations is under 0.002%, the standard is met.

This is classical power system reliability modelling. In the traditional power system, USE arises when extreme demands are experienced in conjunction with simultaneous random failures of large conventional generators.

The transitioning power system reduces this risk but introduces new ones. A future source of USE is renewable energy droughts that extend beyond the energy limits of batteries and hydro. This presents new challenges for reliability modellers.

There is a view that the form of the reliability standard must also change, which has triggered a review conducted by the NEM’s Reliability Panel. The prior assumption is that reliability events will be rarer, but when they occur, they will be more severe. This is described as “tail risk”.

The Panel is contemplating new forms of standard that attempt to limit these extremes, but this necessarily implies a more conservative standard, costing customers more. The AEC is pleased to publish a report by Endgame Economics that theoretically contemplates the appropriateness of the existing standard in the transitioning power system.

Their conclusion is that the existing simple USE output form approach remains the best approach for the future power system, just as it is in the current power system. It is superior to the alternatives being considered.

We take a brief look at the report.

Tail Events

Endgame discuss the shift from a capacity-constrained to energy-constrained system. The NEM used to run out of capacity in the afternoon of extremely hot days, caused by a severe spike in demand that was at least quite short-lived. A future power system with lots of solar generation and battery storage is well placed to address this capacity challenge. Similarly, the risks caused by multiple random failures of very large units will decrease as the coal fleet declines.

However, this is being replaced with the new challenge of extended Variable Renewable Energy (VRE) droughts. To manage these, the limiting factor will be the “depth” of other backup sources, such as energy held in batteries, hydro dams and even diesel tanks.

An assumption then develops that these energy constrained events will be less frequent than the capacity events of old, but when they happen, their impacts on customers will be more severe.

Figure 1 Tail of USE events is forecast to grow

Source: Electricity Statement of Opportunities August 2019

This assumption is to be tested with sophisticated modelling by the Panel. Whilst this will be very interesting, Endgame conclude that even if the assumption is proved correct, it doesn’t have implications for the form of the reliability standard, which currently uses only the average of the distribution.

How do customers experience reliability shortfalls

Reliability events are both the rarest form of interruption experienced by customers, and when they are experienced, they are arguably the least impactful upon customers of all forms of interruption.

Figure 2: Relative causes of customer interruptions actually experienced by customers

Source: AEMC

They are rare because the other forms are, in relative terms, so common.  For example, a car accident knocking down a power pole, for which there are no realistic options to materially address. Although much rarer than distribution events, security events (disturbances in the transmission grid such as the Queensland 2021 event), have always caused more USE than reliability events and will continue to do so.

These other forms of customer interruption are sensibly separated from the reliability standard because addressing them requires entirely different actions.

The consequences of reliability events are also less impactful than these other forms of interruption because they take the form of rotational load shedding, where, under AEMO instruction, networks rotate customers off supply for periods of 30-60 minutes. In a severe reliability event, such as a hypothetical “tail risk” event, more customers will experience a period of load shedding, but individual customers’ experiences will not change.

The existing form of the reliability standard allows the Panel to determine a figure that balances the cost of building more supply against the USE multiplied by the “Value of Customer Reliability”. This figure is determined by the Australian Energy Regulator (AER) as the average harm caused to customers by a blackout.  

The perhaps unintuitive conclusion is that even if we do get a very severe event, while media concerns will be raised, the actual impact on customers will not necessarily be exceptional. Firstly, over the year, in magnitude it will still be swamped by other interruptions, and secondly, whilst the pain will be experienced by many customers, the individual pain experienced is not great.

Considering this, a standard that averages USE across all scenarios, including the tails, remains the correct approach.

The marginal cost of a blackout goes down with time

Notwithstanding the above, the Panel’s Issues Paper took the view that during very long tail events some customers might experience longer rotations, although the mechanism for this was not provided. Endgame have addressed this possibility by showing considerable evidence, from the AER and others, that the marginal value of customer reliability goes down, not up, with extended blackouts.

An extended interruption is obviously very unfortunate, but interestingly a 3 hour interruption is less than three times as inconvenient to a customer than a 1 hour interruption. If it were true that tail events led to longer outages, then it would seem to suggest the long-term economic harm to customers might actually be smaller than that experienced with conventional sub 1 hour rotational load shedding.

This then challenges the tail risk-averse assumption behind the proposals for a new reliability standard form.

Figure 3 Value of customer reliability for residential customers by duration

Source: Endgame Economics

Other forms of the standard

The panel is considering a strawman alternative which applies a Conditional Value at Risk (CVaR) component to the standard. This is presented on an assumption that customers as a whole exhibit risk aversion. Endgame have identified that it fails to engage with the experience of individual customers. Risk aversion depends upon how the rotational load shedding is actually allocated to individual customers and it is wrong to consider the whole load as if it were a single customer being shed.

If it were introduced, it would be very complex to apply and would most certainly require a more conservative system with greater reserve margins, implying above optimum costs to consumer.

AEMO were concerned by the complexity of the strawman, so they have proposed a particularly blunt alternative. They suggest simply discarding the lower 90% of simulations when determining the average. As these simulations have negligible load shedding, AEMO are effectively proposing a standard ten times as conservative as the economic optimum.

Endgame see that as analogous as an insurance company charging an annual premium on the basis of a one-in-ten year flood occurring every year.

Interestingly AEMO suggest that their simple approach gives a similar outcome to the more complex CVaR approach, indicating that that is also inefficiently conservative.

The AEC hopes the Reliability Panel considers Endgame’s considerations in detail before changing from the existing, well-understood and customer focussed form of the Reliability Standard.