Multi-level Reinforcement Learning for flow control
Lead Research Organisation:
Heriot-Watt University
Department Name: Sch of Energy, Geosci, Infrast & Society
Abstract
Flow control is the process of targeted manipulation of fluid flow fields to accomplish a prescribed objective (e.g. reduce drag). Flow control uses information from the flow (provided by sensors) to adapt to incoming perturbations and adjust to changing flow conditions. General flow control is a largely unsolved mathematical problem appearing in many industries, including automotive, aerospace and environmental subsurface flow problems. The missing ingredient for turning flow control into a practical tool is the development of general flow control algorithms that can handle the following: (a) uncertainties in the system perturbations (e.g. the speed and direction of the perturbation), (b) uncertainties in the flow model parameters, (c) sparsity of the observations (i.e. partial and noisy observations) (d) modelling errors due to discretization and parameter upscaling.
In this proposal, Reinforcement Learning (RL) algorithms will be utilized to learn general flow control polices using reliable simulated flow environments. From an application point of view, the developed mathematical techniques address flow control in two applications: (a) increasing energy efficiency in transportation trucks by flow control of incompressible Navier-Stokes flow past an obstacle and (b) safe and efficient storage of anthropogenic carbon dioxide (CO2) in deep geological formations using flow control in a Darcy-type subsurface flow. For the first application, road freight transportation accounts for approximately 5% of the UK's carbon footprint and flow control to reduce the aerodynamic drag could significantly improve the fuel efficiency, for example a 15% reduction in drag is equivalent to about 5% in fuel savings. For the CO2 storage application, the produced CO2 by human activities, for example from a power stations or an energy-intensive industries, could be injected into deep saline aquifers as a possible mitigation strategy to reduce anthropogenic emissions of carbon dioxide into the atmosphere. The control of injection strategies in the subsurface storage sites, given the inherent uncertainties in the subsurface properties, would minimize the risk of leakage while maximising the storage capacity.
In this proposal, Reinforcement Learning (RL) algorithms will be utilized to learn general flow control polices using reliable simulated flow environments. From an application point of view, the developed mathematical techniques address flow control in two applications: (a) increasing energy efficiency in transportation trucks by flow control of incompressible Navier-Stokes flow past an obstacle and (b) safe and efficient storage of anthropogenic carbon dioxide (CO2) in deep geological formations using flow control in a Darcy-type subsurface flow. For the first application, road freight transportation accounts for approximately 5% of the UK's carbon footprint and flow control to reduce the aerodynamic drag could significantly improve the fuel efficiency, for example a 15% reduction in drag is equivalent to about 5% in fuel savings. For the CO2 storage application, the produced CO2 by human activities, for example from a power stations or an energy-intensive industries, could be injected into deep saline aquifers as a possible mitigation strategy to reduce anthropogenic emissions of carbon dioxide into the atmosphere. The control of injection strategies in the subsurface storage sites, given the inherent uncertainties in the subsurface properties, would minimize the risk of leakage while maximising the storage capacity.
Organisations
People |
ORCID iD |
| Ahmed Elsheikh (Principal Investigator) |
Publications
Dixit A
(2022)
Robust Optimal Well Control using an Adaptive Multigrid Reinforcement Learning Framework
in Mathematical Geosciences
Dixit A
(2022)
Stochastic optimal well control in subsurface reservoirs using reinforcement learning
in Engineering Applications of Artificial Intelligence
Shams M
(2023)
Gym-preCICE: Reinforcement learning environments for active flow control
in SoftwareX
| Description | We demonstrated that Reinforcement Learning is an effective tool for learning robust control of physical systems especially for flow control problems. We developed statistical algorithms to accelerate the learning (i.e., more energy efficient) and to tackle large scale problem. The developed mathematical techniques are combined with demonstration software packages for reuse by the scientific and engineering community. |
| Exploitation Route | We are developing an open source software package that will be released on Github such that this work could be used by other researchers in all engineering fields. |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Energy,Environment |
| Description | The developed methods are now being investigated to study control of coupled systems (fluid-structure interaction) that could be used in modeling and optimisation of floating wind turbines. We aim to publish the code with few demonstration to enable a wide adoption of these techniques by the applied engineering community. |
| First Year Of Impact | 2023 |
| Sector | Aerospace, Defence and Marine,Energy,Environment |
| Impact Types | Economic |