Failing to incorporate renewable energy sources effectively into power networks can create serious issues around energy pricing and forecasting. Some neural networks can mitigate renewables’ intermittency, but require the right expertise and data.
Renewables: viable, but volatile
The use and production of renewable power is on the rise. Anthropogenic climate change is now an established phenomenon, and several governments are working to reduce their dependence on fossil fuels and meet climate-related power-generation goals (subject to various geopolitical factors). Meanwhile, the technology behind renewable power generation continues to advance rapidly. Thanks to increasingly efficient power-generation systems, more power can now be generated at lower cost, and as demand for the technology increases, hardware costs are likely to fall too. Improvements to energy storage technology also help to make renewable power more viable.
But while growth in renewables looks set to continue, certain nuances of these energy sources could impede their success. Increasing the proportion of energy derived from iRES links energy production to volatile factors that cannot be controlled. This can reduce the accuracy of short-term pricing, which is important for scheduling conventional power output, and for hedging.
Power from conventional sources – like coal and natural gas – can be scheduled: within reasonable parameters, power stations can increase or decrease their output to meet demand. iRES differ in this regard, because rather than being scheduled, their output can only be predicted. In addition, demand for electrical power is relatively inelastic. Usage doesn’t generally vary with price, and consumers tend to use whatever they need regardless of unit cost (they still bear the cost of price variance, as an added cost from energy providers). Because consumer usage does not act as a buffer for energy consumption (i.e., they don’t tend to buy more when it’s cheap, and less when it’s expensive), there is more pressure on producers to generate power that matches usage as closely as possible.
This is a challenge, because power generation from iRES can be highly intermittent.
- Interday intermittency. Power from iRES on a given day is not stable. This can be partially mitigated by diversifying the power sources in networks (either by geography or by type of power source). Adding more stable power sources2 – or those with flexible output – can also help to smooth out interday variability.
- Intraday intermittency. Power generated from solar sources (and, to a lesser extent, wind) exhibits diurnal patterns(i). For solar power in particular, this pattern – displayed by the ‘duck curve’3 (so called because of its shape: see Figure 1) – is approximately the inverse of power demand. In other words, power generation is high during the day, but demand tends to peak in the morning and evening (though this varies depending on climate and latitude). Even intraday patterns based on averages are highly volatile, as power outputs are tied to highly variable climatic conditions, which are themselves difficult to predict. While large grids with diverse sources of power generation will go some way to damping this hour and sub-hour volatility, they are not enough to eliminate it completely.
Figure 1: The 'duck curve', showing peaks in power demand that do not correlate to iRES outputs
Source: Chartis Research
As iRES account for larger proportions of total power generation, their intermittency has been shown to cause increased variation in short-term wholesale energy prices(ii), reducing the accuracy of price forecasting. This introduces significant price risk into the market(iii). More accurate price forecasts can alleviate some of this risk, allowing conventional power generators to manage their output more effectively, and providing more accurate figures for hedging purposes.
ANNother way to forecast
But forecasting power prices for a given time period is a challenge, even for entirely conventional power networks. Producers and other traders must predict granular time series based on multiple factors, many of them complex weather-related factors (such as localized humidity, air pressure, and cloud cover). To accomplish this, some firms are employing machine learning (ML) methods, of varying viability(iv,v). Among the more promising are artificial neural networks (ANNs) – computing systems that can learn based on example data, rather than just specific parameters4. Increasingly these are being used to forecast short-term power prices – with positive results.
ANNs can forecast prices based on a combination of inputs:
- Historical data (covering price, load and weather) is used to train, and subsequently to test, a neural network. To avoid issues with overfitting5, separate datasets for training and testing are critical. Firms that implement neural networks for forecasting may also require considerably more data.
- Load data. This includes: forecast data on weather and temperature; data about calendar days on which unusual network loads have historically been seen (or on which unusual loads are otherwise expected); and data from power meters.
- iRES power-generation data. This relies mostly on predicted weather patterns, and can improve the accuracy of price forecasting, particularly in networks where iRES penetration is high. (In networks with low iRES penetration, there is unlikely to be a meaningful correlation between iRES output variance and power prices).
While ANNs can be used to forecast prices, there is definitely room for improvement. A potential solution to the accuracy problem (or at least one beneficial course of action) is to incorporate iRES generation forecasts and data into price forecasts, to reduce error rates(vi).
Incorporating iRES generation forecasts can be effective, but it raises additional concerns. In essence, errors in power-generation forecasts can translate into errors in price forecasts(vii), so must be kept to a minimum.
As several studies have shown(viii), neural networks can also be effective in forecasting iRES power generation. Indeed, used in this way neural networks can provide a method that is accurate enough to allow producers to reduce the storage and standby capacity they need. This means that conventional power sources (such as coal and natural gas) no longer need as much contingency built into their production – increasing their operational efficiency. Price volatility is then effectively reduced, because the maximum power deficits or surfeits conventional generators must bear are lowered.
The importance of memory
Within neural network approaches, Long Short-Term Memory (LSTM) networks have shown particular promise, considerably outperforming more basic ANNs in situations where power output fluctuates strongly throughout the day(ix). LSTMs are a subset of recurrent neural networks (RNNs), which themselves differ from more basic ANNs in that they have a ‘memory’ component – previous states in a sequence can have an influence on future values. However, RNNs can only retain a ‘memory’ in the short term, and are increasingly unable to learn how to use the memory as it becomes more distant. LSTMs have a much longer memory, due to a ‘cell state’ that allows distant network outputs to influence current/future outputs, where appropriate.
By using LSTM approaches in forecasting power price, load and iRES power generation, producers can influence predicted values with a number of factors:
- Recent states in time/sequence (this represents the ‘short term’ of LSTM).
- More distant states in time/sequence (representing the ‘long term’ of LSTM).
- Data inputs (e.g., weather, time, day of the week, etc.).
- Parameters of the methodology (either predefined, or learned through training).
Essentially, for each value in a sequence (such as predictions of how much power will be generated in a sequence of given time periods), the LSTM determines several critical components:
- What new information could be influential in future, and thus should be retained (through ‘input gates’).
- Stored information that is no longer influential and which should be forgotten (through ‘forget gates’).
- The components of the cell memory that can currently influence outputs (through ‘output gates’).
This combination of standard neural network functionality and long-term memory can be highly effective, albeit at the cost of increased complexity. And to address this extra complexity, firms will have to engage more expert staff.
An opportunity – with the right expertise and data
Most ANN approaches can offer significant benefits for traders in multiple industry areas, but they are not without their challenges. Users of ANNs will need additional expert staff to develop and manage these new approaches, while firms’ data requirements will likely increase, possibly substantially. Users need datasets to train and test the models, followed by up-to-date input parameters (which can include large, granular time-series datasets).
For those firms trading power products that are able to construct and use neural networks in-house, these challenges present an opportunity. However, while many trading firms are more than capable of implementing ANNs themselves (aided by libraries, depending on the chosen programming language), many will also look for help from third parties. Data vendors are in a strong position, because the data-intensive training and testing stage is an integral part of ANN implementation. So to forecast short-term power prices, future load, or iRES output using neural networks, traders will have to engage with a broad range of data providers.
ANNs represent a good way to ensure accurate short-term power pricing that can account for the intermittency of iRES. There is a caveat, however: their value in mitigating renewable-specific price risk varies according to iRES penetration – areas with heavily entrenched conventional fuel sources may not be concerned with the effects of intermittent renewable power for some time.
1. Although hydroelectric and tidal energy sources are also linked to weather, they are more stable than wind and solar.
2. Stable energy sources can be conventional, such as gas power plants, or more stable renewable sources, like hydroelectric facilities.
3. The duck curve consists of total load in a network, minus the iRES output, so it shows the power needed from conventional (and stable renewable) sources. The two peaks represent high energy requirements that cannot currently be met through iRES (though the duck curve is traditionally associated with solar-focused systems).
4. Some parameters ('hyperparameters'), however, are still set beforehand, rather than being inferred from training.
5. 'Overfitting' is a model's tendency to fit its analysis too closely to the details of a particular dataset.
i Graabak, I., & Korpås, M. (2016). Variability characteristics of European wind and solar power resources – A review. Energies, 9(6), 449.
ii Johnson, E. P., & Oliver, M. E. (2019). Renewable generation capacity and wholesale electricity price Variance. The Energy Journal, 40(5).
iii Kulakov, S., & Ziel, F. (2019). The impact of renewable energy forecasts on intraday electricity prices. arXiv preprint arXiv:1903.09641.
iv Mujeeb, S., Javaid, N., Ilahi, M., Wadud, Z., Ishmanov, F., & Afzal, M. K. (2019). Deep long short-term memory: a new price and load forecasting scheme for big data in smart cities. Sustainability, 11(4), 987.
v Lago, J., De Ridder, F., & De Schutter, B. (2018). Forecasting spot electricity prices: deep learning approaches and empirical comparison of traditional algorithms. Applied Energy, 221, 386-405.
vi Aineto, D., Iranzo-Sánchez, J., Lemus-Zúñiga, L. G., Onaindia, E., & Urchueguía, J. F. (2019). On the influence of renewable energy sources in electricity price forecasting in the Iberian market. Energies, 12(11), 2082.
vii Kiesel, R., & Paraschiv, F. (2017). Econometric analysis of 15-minute intraday electricity prices. Energy Economics, 64(C), 77-90.
viii Sharifzadeh, M., Sikinioti-Lock, A., & Shah, N. (2019). Machine-learning methods for integrated renewable power generation: A comparative study of artificial neural networks, support vector regression, and Gaussian Process Regression. Renewable and Sustainable Energy Reviews, 108(C), 513-538.
ix Lee, D., & Kim, K. (2019). Recurrent neural network-based hourly prediction of photovoltaic power output using meteorological information. Energies, 12(2), 215.
- The risk of clean air
(September 2019, Chartis)
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
You are currently unable to print this content. Please contact email@example.com to find out more.
You are currently unable to copy this content. Please contact firstname.lastname@example.org to find out more.
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Printing this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email email@example.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Copying this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email firstname.lastname@example.org
Voice of the CTO: Enterprise architecture aims to tie together a bank’s overall infrastructure, systems and workflows. While easily said, bank technologists explain why it’s harder in practice.
The growing sophistication of behavioral models: a challenge and an opportunity in ALM, liquidity and balance sheet modeling
A collaborative article by Chartis and Oracle
Voice of the CTO: As banks strive to take advantage of cutting-edge tools, the basics of proper data management and governance are too often overlooked. Banks are learning that a CDO can help gain favor with the C-suite.
A collaborative article by Chartis and NICE Actimize
Voice of the CTO: While generative AI is today’s hot topic, bank technologists explain how they have spent years planning for this moment. But skeptics remain.
The focus on stress tests as a way to protect against shocks to the financial system intensified after the financial crisis, and stress tests continue to evolve to embody new and emerging risks. For many banks in the US, stress testing offers a way to…
Voice of the CTO: In part two of a five-part series, bank technologists explain where firms go wrong when trying to modernize their tech stacks and manage technical debt.
- The growing sophistication of behavioral models: a challenge and an opportunity in ALM, liquidity and balance sheet modeling
- RiskTech100 Winner's Spotlight: Mirai
- Breaking down the walls: Enterprise architecture gets its day in the sun
- Surveillance success: tackling the challenges in communications monitoring
- Exploring the future of GRC: integration and innovation
- Elephant in the room: balancing AI innovation with cloud projects