commit ead45732c3d9d92748169517f74d91d9e4ee26f5 Author: Zuul Date: Sat Oct 10 14:40:41 2020 +0000 Update git submodules * Update nova from branch 'master' - Merge "libvirt: Increase incremental and max sleep time during device detach" - libvirt: Increase incremental and max sleep time during device detach Bug #1894804 outlines how DEVICE_DELETED events were often missing from QEMU on Focal based OpenStack CI hosts as originally seen in bug #1882521. This has eventually been tracked down to some undefined QEMU behaviour when a new device_del QMP command is received while another is still being processed, causing the original attempt to be aborted. We hit this race in slower OpenStack CI envs as n-cpu rather crudely retries attempts to detach devices using the RetryDecorator from oslo.service. The default incremental sleep time currently being tight enough to ensure QEMU is still processing the first device_del request on these slower CI hosts when n-cpu asks libvirt to retry the detach, sending another device_del to QEMU hitting the above behaviour. Additionally we have also seen the following check being hit when testing with QEMU >= v5.0.0. This check now rejects overlapping device_del requests in QEMU rather than aborting the original: https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b This change aims to avoid this situation entirely by raising the default incremental sleep time between detach requests from 2 seconds to 10, leaving enough time for the first attempt to complete. The overall maximum sleep time is also increased from 30 to 60 seconds. Future work will aim to entirely remove this retry logic with a libvirt event driven approach, polling for the the VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED and VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED events before retrying. Finally, the cleanup of unused arguments in detach_device_with_retry is left for a follow up change in order to keep this initial change small enough to quickly backport. Closes-Bug: #1882521 Related-Bug: #1894804 Change-Id: Ib9ed7069cef5b73033351f7a78a3fb566753970d diff --git a/nova b/nova index 2745e68..745f835 160000 --- a/nova +++ b/nova @@ -1 +1 @@ -Subproject commit 2745e685376abbc4c32516837f6074a3de23aa24 +Subproject commit 745f835e9d8931261501747130165824334abb4b