Interpretation and analysis of deep reinforcement learning driven inspection and maintenance policies for engineering systems

Abstract

In the wake of recent Deep Reinforcement Learning (DRL) breakthroughs in various fields, DRL-based approaches have also been successfully developed for the management of engineering systems, reaching performance that significantly exceeds conventional and state-of-the-art decision strategies. By mapping a sufficient statistic, namely a dynamically updated belief state, to the current optimal action and by relying on dynamic programming principles, Partially Observable Markov Decision Processes (POMDPs) are able to accommodate observational and other uncertainties, overcoming the computational complexities associated with the exponential growth of the policy space in time. POMDPs can be efficiently solved via point-based algorithms, up to medium-scale state space settings, whereas for large-scale state, action, and observation spaces, POMDPs can be effectively integrated with DRL through parametrization of the policy and/or value function with artificial neural networks.

While the benefits offered by DRL policies have been clearly demonstrated for a wide range of data collection and intervention planning engineering applications in terms of optimality and scalability, interpretability by decision-makers (e.g., operators, designers, other stakeholders) accustomed to traditional calendar- and/or condition-based policies can be challenging. In this work, we thoroughly describe the fundamentals of DRL-based management policies, propose simulation-based methods for facilitating their interpretation, and investigate their intrinsic adaptability and safety properties. Through a characteristic inspection and maintenance planning setting featuring a k-out-of-n structural system with components subject to stochastic fatigue deterioration, we describe POMDP-DRL management strategies, together with specific implementation and interpretation details.

In addition, we are looking into the effects of manual user interventions at the time of deployment on the action sequence prescribed by the learned DRL policy. It is observed that based on the fact that POMDP-DRL driven policies are defined as a function of the current belief state, if the decision-maker opts for an alternative, even suboptimal, action other than the one suggested by the DRL-based policy, the belief state will be accordingly updated and can still be used as input for the remainder of the planning horizon, without any requirements for network retraining. Further, relevant constraints can be formulated to ensure that certain safety and/or budget restrictions are respected.

Overall, with this investigation we aim to analyze and demonstrate the flexibility, safety, and interpretability of POMDP-DRL based policies, towards accelerating adoption in real-world settings and improving the understanding of AI-driven decisions for engineering systems management.