PhD Studentship in Multi-agent Reinforcement Learning based Cyber-Physical Networked Control for Renewable Energy Fed Smart Grids under Cyber-Attacks

About the Project

The future power grid will be a highly complex cyber-physical system, integrating multiple distributed energy resources (DERs) such as solar, wind, marine, and bioenergy alongside conventional synchronous generators. Efficient management of energy storage devices, including batteries, fuel cells, supercapacitors, hydrogen storage, and plug-in hybrid electric vehicles (PHEVs), will be crucial for ensuring grid stability and economic operation in future high, medium transmission and low voltage power distribution networks. This integration introduces significant challenges in voltage and frequency regulation, energy scheduling, and overall smart grid system optimization. Moreover, such complex interconnections between power system dynamics, communication networks, and information technology increases the grid’s exposure to cyber-attacks, which can compromise measurement signals, disrupt control commands, or induce model or data-driven instability. 

This project aims to develop a robust multi-agent reinforcement learning (MARL) framework for cyber-physical networked fault-tolerant control of renewable energy-fed smart grids under adversarial conditions [6]-[9]. Multiple autonomous agents will represent distributed generators, storage units, and controllable loads, each learning optimal control policies through independent interaction with the environment. The MARL formulation will address coordination problems, conflict resolution, and cooperative/competitive behaviour among cyber-physical actors/agents while optimizing for their local and global objectives such as minimizing generation cost, reducing emissions, maintaining smart grid voltage stability indicators and designing fault tolerant control strategies. Recent advances in reinforcement learning (RL) show promise for real-time energy management of microgrids, battery scheduling, and handling uncertain renewable generation without relying on forecasted data [1]-[5]. Studies reveal that parametric tuning of RL algorithms such as state and action discretization, exploration parameters, and decision intervals significantly affects algorithmic convergence and system performance. Distributional RL techniques, such as quantile regression and prioritized experience replay, improve robustness by modelling the full reward distribution rather than its expectation, making them suitable for environments with high stochasticity and tariff variability. Furthermore, hybrid offline/online and dual-layer RL strategies demonstrate superior performance by combining long-horizon forecasts with real-time corrections, reducing both convergence time and operating costs [1]-[5]. Building on these previous findings, this PhD project will design a new MARL architecture that incorporates model uncertainty, cyber-attack scenarios, and network reconfiguration events. Agents will be trained to detect and respond to false data injection, denial-of-service (DoS), and topology attacks through adversarial training and robust policy learning [8]-[10]. This approach will leverage deep neural networks to approximate value functions and policies, enabling scalability to high-dimensional state-action spaces. A key innovation will be the integration of distributed federated learning, where agents share partial information to enhance resilience while preserving data/model privacy and communication efficiency. The project will also consider economic dispatch and energy trading, treating PHEVs as both loads and distributed generators. Game-theoretic optimization will be embedded within the MARL formulation to handle strategic interactions in energy markets for negotiation amongst actors. The developed algorithms will be validated using simulation testbeds and simple hardware-in-the-loop microgrid setups with battery storage.

Overall, this research will advance the state of the art in resilient, data-driven control of cyber-physical power systems, enabling secure, stable, & efficient operation of future smart grids.  

Project specific entry requirements: The applicant should have a good Master’s degree in Mathematics, Statistics, Computer Science, Engineering, Physics, with good programming, analytical problem-solving, writing and presentation skills.  

Please direct project specific enquiries to: Saptarshi Das, Email: s.das3@exeter.ac.uk 

Advert information