Multi-level Reinforcement Learning for flow control
Lead Research Organisation:
Heriot-Watt University
Department Name: Sch of Energy, Geosci, Infrast & Society
Abstract
Flow control is the process of targeted manipulation of fluid flow fields to accomplish a prescribed objective (e.g. reduce drag). Flow control uses information from the flow (provided by sensors) to adapt to incoming perturbations and adjust to changing flow conditions. General flow control is a largely unsolved mathematical problem appearing in many industries, including automotive, aerospace and environmental subsurface flow problems. The missing ingredient for turning flow control into a practical tool is the development of general flow control algorithms that can handle the following: (a) uncertainties in the system perturbations (e.g. the speed and direction of the perturbation), (b) uncertainties in the flow model parameters, (c) sparsity of the observations (i.e. partial and noisy observations) (d) modelling errors due to discretization and parameter upscaling.
In this proposal, Reinforcement Learning (RL) algorithms will be utilized to learn general flow control polices using reliable simulated flow environments. From an application point of view, the developed mathematical techniques address flow control in two applications: (a) increasing energy efficiency in transportation trucks by flow control of incompressible Navier-Stokes flow past an obstacle and (b) safe and efficient storage of anthropogenic carbon dioxide (CO2) in deep geological formations using flow control in a Darcy-type subsurface flow. For the first application, road freight transportation accounts for approximately 5% of the UK's carbon footprint and flow control to reduce the aerodynamic drag could significantly improve the fuel efficiency, for example a 15% reduction in drag is equivalent to about 5% in fuel savings. For the CO2 storage application, the produced CO2 by human activities, for example from a power stations or an energy-intensive industries, could be injected into deep saline aquifers as a possible mitigation strategy to reduce anthropogenic emissions of carbon dioxide into the atmosphere. The control of injection strategies in the subsurface storage sites, given the inherent uncertainties in the subsurface properties, would minimize the risk of leakage while maximising the storage capacity.
In this proposal, Reinforcement Learning (RL) algorithms will be utilized to learn general flow control polices using reliable simulated flow environments. From an application point of view, the developed mathematical techniques address flow control in two applications: (a) increasing energy efficiency in transportation trucks by flow control of incompressible Navier-Stokes flow past an obstacle and (b) safe and efficient storage of anthropogenic carbon dioxide (CO2) in deep geological formations using flow control in a Darcy-type subsurface flow. For the first application, road freight transportation accounts for approximately 5% of the UK's carbon footprint and flow control to reduce the aerodynamic drag could significantly improve the fuel efficiency, for example a 15% reduction in drag is equivalent to about 5% in fuel savings. For the CO2 storage application, the produced CO2 by human activities, for example from a power stations or an energy-intensive industries, could be injected into deep saline aquifers as a possible mitigation strategy to reduce anthropogenic emissions of carbon dioxide into the atmosphere. The control of injection strategies in the subsurface storage sites, given the inherent uncertainties in the subsurface properties, would minimize the risk of leakage while maximising the storage capacity.
Organisations
Publications
Dixit A
(2022)
Stochastic optimal well control in subsurface reservoirs using reinforcement learning
in Engineering Applications of Artificial Intelligence
Dixit A
(2022)
Robust Optimal Well Control using an Adaptive Multigrid Reinforcement Learning Framework
in Mathematical Geosciences
Shams M
(2023)
Gym-preCICE: Reinforcement learning environments for active flow control
in SoftwareX
| Description | This project explored how artificial intelligence, specifically a technique called reinforcement learning (RL), can be used to optimize the control of complex physical systems, such as fluid flows or CO2 injection in deep geological formations. Reinforcement learning is a type of AI where a computer learns by interacting with a system, receiving rewards for desirable outcomes and penalties for undesirable ones. Over many trials, the AI learns a control policy that maximizes the rewards, essentially learning by trial and error. The key innovations in the projects are: 1- Developing software tools to make it easier to apply RL to fluid dynamics problems 2- Handling uncertainty in the physical properties of the system being controlled 3- Making the learning process computationally efficient, even for complex PDE models 4 - Proposing general frameworks that could be applied to a wide range of PDE-based control problems By advancing these aspects of RL, the authors aim to bring the power of AI to optimize complex physical systems in the real world, such as improving the efficiency of wind turbines, aircraft designs, subsurface utilization in CO2 storage, and more. The ultimate goal is to create AI controllers that can autonomously manage these systems in optimal ways. |
| Exploitation Route | 1- Industry adoption: The techniques developed during this project could be applied by companies in relevant industries, such as wind energy, automotive or aerospace. By using reinforcement learning for optimal control, they could potentially improve efficiency, reduce costs, and increase automation in their operations. 2- Further research and development: there's still more work to be done to fully realize the potential of RL for PDE-based control. Future research could build on these ideas, exploring new applications, improving the algorithms, or integrating with other techniques like model predictive control. 4- Expansion to other domains: While this project focused on fluid dynamics and subsurface flows, the general principles could be extended to other areas involving PDE-based models, such as heat transfer, structural mechanics, or chemical reactions. Researchers in these fields could adapt and apply the techniques to their specific problems. 5- Development of commercial software and tools: The Gym-preCICE software developed during this project is a good example of making advanced mathematical/AI techniques more accessible. Further development of user-friendly software, libraries, and tools could accelerate adoption by researchers and practitioners. 6- Education and training: As these AI techniques become more prevalent, there will be a growing need for professionals who understand and can apply them. Universities and industry training programs could incorporate this material into their curricula to prepare the workforce for the future of autonomous control and optimization. |
| Sectors | Aerospace Defence and Marine Chemicals Digital/Communication/Information Technologies (including Software) Energy Environment |
| URL | https://ai4netzero.github.io/rl_project/ |
| Description | The developed methods are now being investigated to study control of coupled systems (fluid-structure interaction) that could be used in modeling and optimisation of floating wind turbines. We aim to publish the code with few demonstration to enable a wide adoption of these techniques by the applied engineering community. |
| First Year Of Impact | 2023 |
| Sector | Aerospace, Defence and Marine,Energy,Environment |
| Impact Types | Economic |
| Description | Enabling CO2 capture and storage using AI |
| Amount | £1,790,579 (GBP) |
| Funding ID | EP/Y006143/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2023 |
| End | 03/2025 |
| Title | Gym-preCICE |
| Description | Gym-preCICE is a Python preCICE adapter fully compliant with Gymnasium (also known as OpenAI Gym) API to facilitate designing and developing Reinforcement Learning (RL) environments for single- and multi-physics active flow control (AFC) applications. In an actor-environment setting, Gym-preCICE takes advantage of preCICE, an open-source coupling library for partitioned multi-physics simulations, to handle information exchange between a controller (actor) and an AFC simulation environment. The developed framework results in a seamless non-invasive integration of realistic physics-based simulation toolboxes with RL algorithms. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | The development and integration of Gym-preCICE into the fields of reinforcement learning (RL) and active flow control (AFC) applications have significant implications for both the industrial and academic sectors in the UK, encompassing aspects like research and development (R&D), industrial applications, educational advancements, and collaborative opportunities. In terms of impact, one could list Improved Efficiency and Accuracy: By enabling RL algorithms to interact with detailed simulations, this integration can lead to the discovery of more efficient and accurate AFC strategies, driving innovation in fields where flow dynamics play a crucial role. Cross-disciplinary Research: Gym-preCICE encourages collaboration between computer scientists, engineers, and physicists, fostering a multidisciplinary approach to solving complex problems. Partnerships between Academia and Industry: The practical applications of Gym-preCICE in industrial R&D can stimulate partnerships between universities and companies, leading to joint ventures, internships, and collaborative projects. Positioning the UK as a Leader: By adopting and contributing to advancements like Gym-preCICE, the UK can position itself as a leader in the integration of AI and multi-physics simulations, attracting international students, researchers, and investments. |
| URL | https://github.com/gymprecice/gymprecice |
| Description | International Congress on Industrial and Applied Mathematics (ICIAM 2023 Tokyo) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presented the algorithms & software package developed as a part of this grant. Talk title "Gym-preCICE: Reinforcement Learning Environments for Active Flow Control" |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://iciam2023.org/registered_data?id=02212#05424 |
| Description | Present at the ECMOR Conference, Bergen Norway |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presented a talk on: Robust Well-Production Control Using Surrogate Assisted Reinforcement Learning |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://doi.org/10.3997/2214-4609.202244101 |
| Description | The 9th International Conference on Machine Learning, Optimization, and Data Science |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Mosayeb presented a talk about gymprecice, the software package developed during this project. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://lod2023.icas.cc/ |
| Description | preCICE workshop 2023, Technical University of Munich, Germany Feb 2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Delivered a talk on: Gym-OpenFOAM: An OpenAI Gym environment for active flow control with deep reinforcement learning |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://precice.org/precice-workshop-2023.html |