Five-minute settlement starts

A momentous change in the National Electricity Market (NEM) occurs at midnight 30 September. Energy will no longer be settled in 30-minute slices on the average of six, five-minute dispatch prices. Instead, each five-minute interval will be settled against its own dispatch price (5ms).

Sounds simple? Conceptually perhaps, yet over the last three years hundreds of millions of dollars worth of information technologists’ time was invested to make it happen. It has affected all parts of the industry, from generators, to retailers, to networks, to metering providers, to derivative traders and of course, the Australian Energy Market Operator (AEMO) itself.

Hundreds of industry people have their fingers crossed that the transition occurs without major problems, and the Australian Energy Council is quietly confident that it will.

Sadly the COVID lockdown will prevent most of the 5ms army enjoying a well-deserved celebration of this great achievement – so here, we take a reflective look on how this came to be, why it was changed, and some of the adventures along the way.

What was 30-minute settlement?

Before the advent of markets, there was no great need to measure power flows in small time intervals and most customers’ meters were read quarterly. The most granular revenue quality meters read half-hourly, which were only used for very large customers and the interconnector between state utilities.

However markets require detailed settlement data, and interval meters were rolled out to generators and to progressively smaller customers. 30-minutes was chosen for the settlement interval, being the most sophisticated revenue meter used in the 1990’s.

When the NEM began in 1998 it was recognised that because supply and demand are constantly in flux, frequent rescheduling is required. Five-minutes was chosen for the dispatch interval, limiting the role of Frequency Control Ancillary Services (FCAS) to correct only small variations between these frequent reschedules. But with the settlement interval being 30-minutes, there is an obvious mismatch for the pricing of energy.

Even before market start arguments raged about how to best resolve the mismatch, sophisticated proposals were developed around volume-weighting prices, with compensation schemes for parties harmed by the mismatch. Ultimately the design chose a simple average of the six, five-minute prices. It was accepted this was imperfect but hoped the impact would be low.

Figure 1: Half hour averaging

Source: AEMC

For a base-load generator or flat customer whose operation doesn’t change much within a half-hour, the mismatch really made no difference. But for a peak load generator, or a customer responding to price, it could be quite material. The NEM’s characteristic volatility is key: a price responsive customer might happily consume what it thinks is cheap energy for twenty-five minutes, only to find a spike in the last dispatch interval price brings the half-hour average into the thousands of $/MWh, by which time five-sixths of the interval’s energy has already been consumed.

The mismatch also affected behaviour. If a half-hour begins with a very high price, it is common to see dispatch prices sink all the way to the price floor (-$1,000/MWh) by the end of the half-hour. This is because parties continue to react - by increasing generation or reducing consumption - to the incentives created by the high average settlement price, even after whatever caused the initial high dispatch price has passed.

Figure 2: Extreme example of oscillating prices

Source: AEMC, Existing distortions in the NEM 2017

A rule mandate for 5ms

In the first years of the market, the National Electricity Market Management Company (NEMMCO) was already working on a fix. They proposed that generators and very large customers receive “simulated five-minute settlement” using their Supervisory Control and Data Acquisition (SCADA) meters, while smaller customers would be settled on volume weighted 30-minute prices. Although this was a relatively low-cost approach, at the time NEMMCO still concluded its costs outweighed its benefits. And so the mismatch continued for another two decades.

At the end of 2015 a price-responsive zinc refinery, Sun Metals, who had experienced some of the challenges described above, proposed NEMMCO’s SCADA based design as a NEM Rule change to the Australian Energy Markets Commission (AEMC). They were supported by consumer groups and developers of very fast responding technologies, especially battery storage.

Opponents were generally energy companies heavily invested in the existing design who would face large transitional costs in migrating systems to a new settlement basis.

Opponents also raised concerns about the proposed use of low accuracy SCADA meters. The AEMC responded by changing from the NEMMCO design to an end-to-end 5ms solution obliging thousands of generator, network, and customer half-hourly revenue meters to be reconfigured to read every five-minutes. This approach certainly resolved the quality concerns, but in doing so inflated system costs many times over.

On 28 November 2017 the AEMC concluded to go ahead with their end-to-end solution, but recognising its immensity, provided the longest lead-time of any NEM reform: three years and seven months.

During 2017 a bizarre media narrative developed: if you were “green” and progressive you supported the reform, but the opposed were “brown” and conservative, with an Australia Institute director stating:

“While it’s welcome that the AEMC has finally acknowledged that we need this important rule change, 2021 is an unacceptable delay.”

We now know that AEMO needed every minute of that lead-time plus three months more.

A true mega-project

Reading meters and settling the resulting transactions six times more often is conceptually trivial, yet the change reached deep into every part of the industry.

Figure 3: Electricity market information flows affected by five-minute settlement

Source: AEMO 2017

Most meters weren’t physically replaced but required reprogramming to accumulate more frequently and store more data. Smaller meters required new deeming arrangements to estimate five-minute quantities.

Millions of lines of energy analysis and settlement software were affected. Even generators’ bidding software changed to permit bidding in five-minute blocks. As increasingly more intricacies were uncovered, expedited consequential rule changes were required.

Recognising the immensity of the change, AEMO took leadership and kicked off a major project in 2018. This engaged deeply and constantly with industry and throughout tracked its readiness. AEMO engaged a small army of IT experts and alone incurred costs of $121m.

Figure 4: Initial expectations of AEMO’s project plan

Source: AEMO 2017

Midway through the project the AEC conducted a confidential survey of its members’ costs, which were by then well scoped. It estimated a cost for the competitive sector alone in the order of $200-$300m. On top of these would be similar magnitude costs incurred by networks and metering providers.

In hindsight an AEC 2017 consultancy to the rule change under-estimated systems change costs at $150m across the industry. Similarly the AEMC’s own consultant seemed to miss a zero:

“Although not directly in scope, we provide the AEMC with a comment that the costs of system changes to moving to 5/5 market settlements will be very large, and estimated to be in the tens of millions of dollars.”

Yet at the time even these estimates were challenged by some:

“It is recognised that there will be some costs involved in the transition to five-minute settlement, however we are convinced that some of the cost projections being stated in the public forums considering this rule change were excessive.”

“Uniting Communities is consequently satisfied that there will only be modest real, additional costs for implementing five-minute settlement, most of which will be one-off, set up costs.”

Global settlement

Given this mega project was reworking the industry’s settlement systems, AEMO saw an opportunity to introduce another major reform: global settlement (GS). This ended “settlement by difference” where the original “host” retailer acts as an energy balancing agent for a geographical area. That model had become outdated thanks to the success of retail competition.

AEMO’s initiative of combining the two projects was pragmatic and generally welcomed by stakeholders. Global Settlement takes effect 1 May 2022.

COVID-19

During 2019 the scale of the necessary systems changes was becoming understood across industry. Concerns emerged about readiness for the July 2021 start with some organisations considering proposing a delay via another rule change, noting this would take many months while adding to project uncertainty. By then however AEMO’s 5ms/GS project was in full swing and conducting regular industry readiness testing. This was progressing reasonably satisfactorily until the COVID crisis emerged in March 2020.

In those first weeks of COVID it was entirely unknown how the industry, indeed the entire economy, would continue to operate. The AEC suggested reconsidering the timetables of five major industry reforms, one of which was a possible six-month deferral of 5ms/GS.

In April 2020 AEMO went considerably further than the AEC’s suggestion, submitting an expedited rule change for a 12-month deferral. The AEMC received 45 submissions with varied views. A common concern was that industry would need to keep the army of IT specialists in place for much longer, adding yet more costs to a very expensive project. The AEMC also observed the impact of the COVID crisis on the energy sector was less disruptive than initially feared, and that IT workers were successfully migrating to work-from-home arrangements.

On 9 July 2020, the AEMC determined for a three month delay only, i.e. starting on 1 October 2021. While this was still a challenging date for some organisations, a degree of certainty had been achieved, or so it was thought.

AEMO 2021 problems

One affected organisation whose readiness had never seemed in doubt was AEMO, having operated a well structured and resourced project from early 2018. When surveying industry readiness through 2019 and 2020, they repeatedly assessed themselves with a “green” status.

However AEMO notified the market that its project status was trending downwards in December 2020, and had reached critical red status by May 2021. The delay was due to issues resulting from several projects operating in parallel to the 5ms project, in particular a project to build an enterprise cloud platform that would contain significant improvements in stability, performance and security. As such, the retail platform was unable to be implemented as scheduled at the end of March 2021, eventually pushing back its implementation until mid-June and severely limiting the opportunity for any in-project contingency.

Contingency rule change

While it is not unusual for a major IT project to suffer contingencies and for timetables to slip, a peculiarity of the 5ms/GS project is that its completion timeframes are hard coded into the rules. If AEMO were to miss the date, it would break the rules. Getting rules changed to reflect a missed date is itself subject to statutory timeframes of at least two months. This presents a dilemma.

On 31 July 2021, AEMO did something of a first by lodging a “use in case of emergency” rule change for a further delay to the 5ms commencement date. While AEMO was still reasonably confident at this time that the project would hit 1 October 2021, it went through the administrative process of arguing for another delay, potentially into 2022.

With AEMO’s early problems now resolved, and industry committed to minimising delays in implementation, all stakeholders agreed at the end of August 2021 that the rule change should not proceed further, as a delay would result in greater costs than the risks it sought to mitigate.

The AEMC acted efficiently, notifying the market on 7 September that the rule change would not be made, and the 1 October start date would remain.

When the market switches from 30-minute settlements to 5ms, spare a thought for the gravity of the task that has been undertaken. Mega-projects take time, and delays should be expected. As industry and AEMO breathe a sigh of relief for a minute or two, we move onto the next significant change to the market. Customer switching enhancements also commence at midnight, wholesale demand response kicks off in a few weeks, and the Consumer Data Right for energy is coming soon.

This sector is transitioning rapidly. Don’t listen to those who say it isn’t.