Maintenance Planning for Bridges using Hierarchical Reinforcement Learning

Abstract

The planning of interventions on transportation infrastructures commonly faces multiple challenges, mainly relating to complexity and scale [1]. Those challenges stem from the uncertainties surrounding infrastructures deterioration and the effect of interventions, which are monitored through visual inspections at the element-level [2]. Element-level information represents the primary resource for supporting decision-making at the bridge-level and network-level. Therefore, the formulation of the planning problem has to accommodate the hierarchical nature of information and decisions. Recent developments have relied on reinforcement learning (RL) for solving maintenance planning problems, with the aim to minimize the long-term costs. Nonetheless, existing RL solutions have adopted approaches that often lacked the capacity to scale due to the inherently large state/action spaces [1]. This paper presents a hierarchical decision-making environment for infrastructures, which naturally adapt to the hierarchy of information and decisions in maintenance planning. The hierarchical formulation enables decomposing large state and action spaces into smaller ones, by relying on state abstraction [3]. An example of state abstraction is when the health states of multiple beam elements within a bridge are collectively represented by an overall expected value and variance. The same concept can be applied to provide state representations at higher levels such as, the bridge-level and network-level. Each of the aforementioned levels has an action space, where interdependent maintenance policies can be applied. For example, a bridge-level where the action space is defined as maintain or do nothing, if a policy suggests doing nothing, such action implies that no intervention is applied on all elements within the bridge, without requiring to assess their health states. Furthermore, the deterioration analyses within this new environment are performed using state-space models (SSM), which allows the probabilistic estimation of the deterioration condition along with the deterioration speed [2]. Accordingly, it becomes possible to assess the effect of including the deterioration speed on the decision-making process. The functionality of the environment is demonstrated by learning a maintenance policy for a structural category composed of multiple structural elements.

[1] Andriotis, C.P., & Papakonstantinou, K.G. (2019). Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliability Engineering & System Safety,191,106483.

[2] Hamida, Z., & Goulet, J.A. (2022). A stochastic model for estimating the network‐scale deterioration and effect of interventions on bridges. Structural Control and Health Monitoring,29(4),e2916.

[3] Nachum, O., Gu, S.S., Lee, H., & Levine, S. (2018). Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems,31.