Sunshine and Storms: How Australia Became a World Leader in Predicting the Sun’s Power
Sydney, on a seemingly ordinary Wednesday in late spring last year, experienced a dramatic midday upheaval in its wholesale electricity market. Within the span of just one hour, fortunes were made and lost as the grid grappled with an unexpected shift in energy supply.
Typically, New South Wales’ energy demand follows a predictable pattern. Mornings see a dip as rooftop solar panels generate power for households, reducing their reliance on the main grid. This is often followed by a sharp rise in demand during the late afternoon, culminating in an evening peak. However, on this particular day, around 11 am, the grid deviated sharply from its usual script. The culprit? Clouds.
As Australia increasingly integrates solar energy into its national power mix, the ability to accurately forecast cloud cover and sunlight has become a multi-billion-dollar industry. This sophisticated field, operating at the intersection of artificial intelligence, big data, the energy transition, and complex meteorological systems, is crucial for maintaining grid stability and keeping energy traders on their toes. While many of us may not be directly aware of solar forecasting, its influence is felt daily. The fundamental question it seeks to answer is deceptively simple: will it be sunny? Through a combination of necessity and ingenuity, Australia has emerged as a global frontrunner in providing accurate answers, with its developed forecasting technology now being adopted worldwide.
The Impact of Clouds on Power Prices
That fateful Wednesday, a fast-moving spring thunderstorm, characterised by dense cloud cover, descended upon Sydney. This effectively throttled the output from rooftop solar installations, leading to a sudden surge in demand for electricity from the grid.
“Suddenly, there was far less [rooftop] generation than anticipated,” explained Julian de Hoog, CEO of Solstice AI, a solar forecasting company. “In aggregate, it’s like losing a power plant. And that’s when prices spike.”
The wholesale cost of electricity in NSW skyrocketed, reaching over $20,000 per megawatt-hour, a stark contrast to the monthly average of approximately $75/MWh. These spot prices, updated every five minutes, reflect the immediate balance of supply and demand on the energy grid. For a crucial 20-minute period, anyone with electricity to sell stood to make a significant profit.
Shortly after, the skies cleared, solar generation resumed, and the cooling effect of the storm reduced the demand for air conditioning. The spot price plummeted. Within an hour of the initial spike, the price had crashed to almost negative $1,000 per megawatt-hour, meaning generators had to pay network operators to accept their power. Energy traders, generators, and battery operators experienced dramatic swings in fortune based on their preparedness for this volatile event.
The question then arises: why don’t such highly volatile price events occur more frequently, given that cloudy days are common? The answer lies in the power of solar forecasting.
Answering the Crucial Question: Will It Be Sunny?
More than a decade ago, a PhD student at the Australian National University recognised the grid’s increasing susceptibility to cloud cover as solar power’s contribution grew. In 2011, solar accounted for a meagre 3 per cent of Australia’s generation capacity. However, Nick Engerer predicted that within ten years, solar would be capable of meeting most of Australia’s electricity needs on sunny days.
This foresight led him to co-found Solcast, a Sydney-based company that pioneered “nowcasting” – the precise forecasting of sunlight intensity (irradiance) at five-minute intervals. “When you reach 20 per cent of total energy coming from solar in a given market, forecast errors suddenly become problematic,” Dr Engerer stated.
Unlike conventional weather forecasts that might predict irradiance across a city, solar forecasting aims for hyper-local accuracy, down to areas as small as 500 square metres. Furthermore, these forecasts are delivered in five-minute increments, a level of detail far beyond daily or hourly predictions.
Solcast quickly established itself as Australia’s leading solar forecasting company. Its data is indispensable to the Australian Energy Market Operator (AEMO), which is tasked with ensuring the stability of the nation’s grids. The company’s insights are also relied upon by approximately 80 per cent of network operators responsible for the physical infrastructure (poles and wires) and 80 per cent of generator-retailers, the companies that both produce and sell electricity directly to consumers. Ben King, Solcast’s chief commercial officer, noted that about one-third of utility-scale solar farms in Australia are also clients.
Solcast is now part of a major Norwegian risk management firm, and its data is utilised globally by clients managing over 300 gigawatts of solar projects – a capacity roughly three times that of all power stations, wind farms, and solar farms currently operating in Australia. “There’s a huge amount of solar that’s going to be built globally in the next three decades,” Mr King observed. “Solar forecasting is a core enabling technology for the reliable operation of electricity grids.”
The Risk of Blackouts
While seemingly a niche field, solar forecasting profoundly influences household power costs and the very reliability of the electricity grid. Australia’s wholesale power prices are among the most volatile globally. This volatility is amplified by the increasing integration of intermittent renewable sources like wind and solar, coupled with the growing frequency of extreme weather events.
Higher volatility translates directly into increased power bills, as electricity retailers build in a buffer to hedge against unpredictable spikes in wholesale prices. In extreme scenarios, if network operators cannot balance fluctuating demand with the variable supply from the vast network of solar panels, the entire grid could collapse, leading to widespread power outages. Solar forecasting plays a critical role in mitigating these price spikes. By anticipating a decline in rooftop solar generation well in advance, other power sources can be brought online proactively.
Predicting sunlight is also a lucrative endeavour. In the 2024–25 financial year, a significant portion of revenue generated by battery energy storage systems was concentrated within just two per cent of the year – specifically, on the handful of days when spot prices experienced dramatic spikes. Battery storage operators can maximise their profits by strategically discharging power during these high-price events, provided they can accurately predict them. Conversely, solar generators can incur losses when spot prices turn negative or when they deliver more or less electricity to the grid than contractually obligated, often due to unforeseen cloud cover.
Given the critical importance of solar forecasting to grid stability, one might assume Australia developed this technology purely out of necessity. While necessity was a driving factor, a significant element of good fortune also played a role.
How Australia Became a Solar Forecasting Pioneer
In 2014, Japan launched the Himawari-8, a next-generation weather satellite offering high-resolution, publicly available images at an unprecedented frequency. Crucially, this satellite was positioned to provide optimal coverage over Australia and the Asia-Pacific region. Prior to Himawari-8, the best available weather satellites provided images with pixels measuring 5km across, updated only every 30 minutes. Himawari-8 dramatically improved this, offering 500m resolution with images refreshed every 10 minutes.
“Himawari-8 made cloud tracking possible,” recalled James Luffman, a co-founder and former CEO of Solcast. “We were obsessed with the problem of where the clouds were going.” The Solcast team developed sophisticated algorithms to process this satellite data, cleaning it to distinguish between clouds and other reflective surfaces like sun glints on water or snow. They also devised methods to determine cloud height and opacity, enabling accurate calculations of how much sunlight would reach the ground.
The Australian Renewable Energy Agency provided crucial funding to support the development of this pioneering technology. Three years later, Solcast triumphed over international competitors in a US solar forecasting competition, solidifying its position as a world leader. “Solar started in Australia early, and Australia began facing forecasting challenges very early on,” Mr King commented. “And Australia continues to lead the world in many ways of managing that, simply because there’s so much solar in Australia.”
Nature’s Curveballs: When Prediction Fails
Despite the advancements, nature occasionally throws unexpected curveballs. The inherent unpredictability of atmospheric interactions involving heat and water can still confound even the most advanced forecasting systems, as demonstrated by the recent Sydney spring storm.
Convective clouds, such as those forming thunderstorms, pose a significant challenge for solar forecasting, according to Alex Zadnik, business manager for MetraWeather Australia. “They can form very rapidly,” he noted. “You might have a satellite image from 10 minutes ago, and in the next 10 minutes, you’ve got these big, fluffy clouds appearing over your roof or across the city. Being able to predict when they’re going to form and where they’re going to impact is a real challenge.” The consequences of inaccurate forecasting are escalating as more solar power is integrated into the grid.
Artificial intelligence (AI) is emerging as a key component of the solution. Conventional weather models are typically vast, complex algorithms, often comprising millions of lines of code and requiring the world’s most powerful supercomputers to simulate atmospheric interactions. AI models, conversely, leverage machine learning to predict future weather patterns by identifying trends within historical datasets.
“The big advantages of AI models are that they are radically cheaper than physics models, which are extremely computationally expensive,” Solcast’s Ben King explained. “We are starting to see AI models in the weather space that can achieve performance levels very close to, or even exceeding, those of physics models.”
Dr de Hoog envisions a future where the solar forecasting industry experiences a neat inversion: instead of analysing clouds to predict sunlight, solar panels themselves could be used to detect clouds. The movement of clouds over a city could be tracked by observing successive drops in electricity generation from rooftop panels beneath them. This high-resolution data could then be fed back into solar forecasting systems to improve accuracy. “That’s definitely on the roadmap,” Dr de Hoog confirmed.
