[2026-04-16T04:51:34.860Z] Started by upstream project "anvil/anvil-pipeline/main" build number 289 [2026-04-16T04:51:34.860Z] originally caused by: [2026-04-16T04:51:34.860Z] Push event to branch main [2026-04-16T04:51:35.747Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-04-16T04:51:35.754Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-04-16T04:51:35.920Z] Examining kronosnet/ci-tools [2026-04-16T04:51:35.920Z] Attempting to resolve main as a branch [2026-04-16T04:51:36.080Z] Resolved main as branch main at revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 [2026-04-16T04:51:36.081Z] Selected Git installation does not exist. Using Default [2026-04-16T04:51:36.081Z] The recommended git tool is: NONE [2026-04-16T04:51:36.081Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-16T04:51:36.083Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-16T04:51:36.083Z] Wiping out workspace first. [2026-04-16T04:51:36.103Z] Cloning the remote Git repository [2026-04-16T04:51:36.103Z] Using shallow clone with depth 1 [2026-04-16T04:51:36.103Z] Avoid fetching tags [2026-04-16T04:51:36.103Z] Honoring refspec on initial clone [2026-04-16T04:51:36.103Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-16T04:51:36.103Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@2 # timeout=10 [2026-04-16T04:51:36.112Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-16T04:51:36.112Z] > git --version # timeout=10 [2026-04-16T04:51:36.119Z] > git --version # 'git version 2.47.3' [2026-04-16T04:51:36.119Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-16T04:51:36.126Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T04:51:36.318Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-16T04:51:36.325Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T04:51:36.334Z] Avoid second fetch [2026-04-16T04:51:36.334Z] Checking out Revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 (main) [2026-04-16T04:51:36.334Z] > git config core.sparsecheckout # timeout=10 [2026-04-16T04:51:36.341Z] > git checkout -f 73ed6275b1ab90dc4db382514cbc5070a56e2e90 # timeout=10 [2026-04-16T04:51:36.374Z] Commit message: "Revert "New getUpNodesByLabel() call"" [2026-04-16T04:51:36.381Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-04-16T04:51:36.574Z] Examining kronosnet/ci-tools [2026-04-16T04:51:36.574Z] Attempting to resolve main as a branch [2026-04-16T04:51:36.732Z] Resolved main as branch main at revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 [2026-04-16T04:51:36.732Z] Selected Git installation does not exist. Using Default [2026-04-16T04:51:36.732Z] The recommended git tool is: NONE [2026-04-16T04:51:36.733Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-16T04:51:36.735Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-16T04:51:36.735Z] Wiping out workspace first. [2026-04-16T04:51:36.759Z] Cloning the remote Git repository [2026-04-16T04:51:36.759Z] Using shallow clone with depth 1 [2026-04-16T04:51:36.759Z] Avoid fetching tags [2026-04-16T04:51:36.759Z] Honoring refspec on initial clone [2026-04-16T04:51:36.759Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-16T04:51:36.759Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-04-16T04:51:36.781Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-16T04:51:36.781Z] > git --version # timeout=10 [2026-04-16T04:51:36.808Z] > git --version # 'git version 2.47.3' [2026-04-16T04:51:36.808Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-16T04:51:36.816Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T04:51:37.019Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-16T04:51:37.037Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T04:51:37.066Z] Avoid second fetch [2026-04-16T04:51:37.066Z] Checking out Revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 (main) [2026-04-16T04:51:37.066Z] > git config core.sparsecheckout # timeout=10 [2026-04-16T04:51:37.078Z] > git checkout -f 73ed6275b1ab90dc4db382514cbc5070a56e2e90 # timeout=10 [2026-04-16T04:51:37.114Z] Commit message: "Revert "New getUpNodesByLabel() call"" [2026-04-16T04:51:37.181Z] [Pipeline] Start of Pipeline [2026-04-16T04:51:37.271Z] [Pipeline] node [2026-04-16T04:51:37.273Z] Running on anvil-ci-bm-phy04 in /srv/workspace/anvil/anvil-build-functional-testing [2026-04-16T04:51:37.274Z] [Pipeline] { [2026-04-16T04:51:37.282Z] [Pipeline] stage [2026-04-16T04:51:37.282Z] [Pipeline] { (Prep build env) [2026-04-16T04:51:37.291Z] [Pipeline] script [2026-04-16T04:51:37.292Z] [Pipeline] { [2026-04-16T04:51:37.296Z] [Pipeline] cleanWs [2026-04-16T04:51:37.359Z] [WS-CLEANUP] Deleting project workspace... [2026-04-16T04:51:37.359Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-16T04:51:37.392Z] [WS-CLEANUP] done [2026-04-16T04:51:37.415Z] [Pipeline] lock [2026-04-16T04:51:37.415Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-04-16T04:51:37.415Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-04-16T04:51:37.415Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-04-16T04:51:37.415Z] [Pipeline] { [2026-04-16T04:51:37.417Z] [Pipeline] } [2026-04-16T04:51:37.418Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-04-16T04:51:37.420Z] [Pipeline] // lock [2026-04-16T04:51:37.424Z] [Pipeline] } [2026-04-16T04:51:37.426Z] [Pipeline] // script [2026-04-16T04:51:37.428Z] [Pipeline] } [2026-04-16T04:51:37.430Z] [Pipeline] // stage [2026-04-16T04:51:37.433Z] [Pipeline] stage [2026-04-16T04:51:37.433Z] [Pipeline] { (Run functional testing) [2026-04-16T04:51:37.439Z] [Pipeline] script [2026-04-16T04:51:37.440Z] [Pipeline] { [2026-04-16T04:51:37.458Z] [Pipeline] echo [2026-04-16T04:51:37.458Z] runWithTimeout 360 [2026-04-16T04:51:37.458Z] [Pipeline] timeout [2026-04-16T04:51:37.459Z] Timeout set to expire in 6 hr 0 min [2026-04-16T04:51:37.459Z] [Pipeline] { [2026-04-16T04:51:37.464Z] [Pipeline] sh [2026-04-16T04:51:38.061Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm almalinux-9 enterprise ci all [2026-04-16T04:51:38.061Z] CITBRANCH: main [2026-04-16T04:51:38.061Z] CITHOME: /root/ci-tools [2026-04-16T04:51:38.061Z] CITCMD: ci-setup-anvil-bm-vm [2026-04-16T04:51:38.061Z] CITCMDARGS: almalinux-9 enterprise ci all [2026-04-16T04:51:38.435Z] Genering ssh keys and ssh config [2026-04-16T04:51:38.435Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:51:38.435Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:51:38.435Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:51:38.811Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:51:38.811Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:51:38.811Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-04-16T04:54:50.919Z] Destroying VMs (if any) [2026-04-16T04:54:50.919Z] Warning: Permanently added 'an-striker01' (ED25519) to the list of known hosts. [2026-04-16T04:54:50.919Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-04-16T04:54:50.919Z] This might take a few seconds. [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T04:54:50.919Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T04:54:50.919Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] Manage your connected systems: https://red.ht/connector [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] The following errors were encountered during disconnect: [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] TYPE STEP ERROR [2026-04-16T04:54:50.919Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T04:54:50.919Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T04:54:50.919Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] 1. Register with subscription-manager [2026-04-16T04:54:50.919Z] # subscription-manager register [2026-04-16T04:54:50.919Z] [2026-04-16T04:54:50.919Z] 2. Register with insights-client [2026-04-16T04:54:50.919Z] # insights-client --register [2026-04-16T04:54:50.919Z] This system is currently not registered. [2026-04-16T04:54:53.401Z] Domain 'an-striker01' destroyed [2026-04-16T04:54:53.401Z] [2026-04-16T04:54:53.401Z] Domain 'an-striker01' has been undefined [2026-04-16T04:54:53.401Z] [2026-04-16T04:54:53.401Z] Warning: Permanently added 'an-striker02' (ED25519) to the list of known hosts. [2026-04-16T04:54:53.401Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-04-16T04:54:53.401Z] This might take a few seconds. [2026-04-16T04:54:53.401Z] [2026-04-16T04:54:53.401Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T04:54:58.512Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T04:54:58.512Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T04:54:58.512Z] [2026-04-16T04:54:58.512Z] Manage your connected systems: https://red.ht/connector [2026-04-16T04:54:58.512Z] [2026-04-16T04:54:58.512Z] The following errors were encountered during disconnect: [2026-04-16T04:54:58.512Z] [2026-04-16T04:54:58.512Z] TYPE STEP ERROR [2026-04-16T04:54:58.512Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T04:54:58.512Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T04:55:02.607Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T04:55:02.607Z] [2026-04-16T04:55:02.607Z] 1. Register with subscription-manager [2026-04-16T04:55:02.607Z] # subscription-manager register [2026-04-16T04:55:02.607Z] [2026-04-16T04:55:02.607Z] 2. Register with insights-client [2026-04-16T04:55:02.607Z] # insights-client --register [2026-04-16T04:55:02.981Z] This system is currently not registered. [2026-04-16T04:55:05.459Z] Domain 'an-striker02' destroyed [2026-04-16T04:55:05.459Z] [2026-04-16T04:55:05.459Z] Domain 'an-striker02' has been undefined [2026-04-16T04:55:05.459Z] [2026-04-16T04:55:05.459Z] Warning: Permanently added 'an-a01n01' (ED25519) to the list of known hosts. [2026-04-16T04:55:05.459Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-04-16T04:55:05.459Z] This might take a few seconds. [2026-04-16T04:55:05.459Z] [2026-04-16T04:55:05.459Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T04:55:09.551Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T04:55:09.551Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T04:55:09.551Z] [2026-04-16T04:55:09.551Z] Manage your connected systems: https://red.ht/connector [2026-04-16T04:55:09.551Z] [2026-04-16T04:55:09.551Z] The following errors were encountered during disconnect: [2026-04-16T04:55:09.551Z] [2026-04-16T04:55:09.551Z] TYPE STEP ERROR [2026-04-16T04:55:09.551Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T04:55:09.551Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T04:55:13.641Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T04:55:13.641Z] [2026-04-16T04:55:13.641Z] 1. Register with subscription-manager [2026-04-16T04:55:13.641Z] # subscription-manager register [2026-04-16T04:55:13.641Z] [2026-04-16T04:55:13.641Z] 2. Register with insights-client [2026-04-16T04:55:13.642Z] # insights-client --register [2026-04-16T04:55:14.439Z] This system is currently not registered. [2026-04-16T04:55:16.921Z] Domain 'an-a01n01' destroyed [2026-04-16T04:55:16.921Z] [2026-04-16T04:55:16.921Z] Domain 'an-a01n01' has been undefined [2026-04-16T04:55:16.921Z] [2026-04-16T04:55:17.294Z] Warning: Permanently added 'an-a01n02' (ED25519) to the list of known hosts. [2026-04-16T04:55:17.668Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-04-16T04:55:17.668Z] This might take a few seconds. [2026-04-16T04:55:17.668Z] [2026-04-16T04:55:17.668Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T04:55:21.774Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T04:55:21.774Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T04:55:21.774Z] [2026-04-16T04:55:21.774Z] Manage your connected systems: https://red.ht/connector [2026-04-16T04:55:21.774Z] [2026-04-16T04:55:21.774Z] The following errors were encountered during disconnect: [2026-04-16T04:55:21.774Z] [2026-04-16T04:55:21.774Z] TYPE STEP ERROR [2026-04-16T04:55:21.774Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T04:55:21.774Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T04:55:24.995Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T04:55:24.995Z] [2026-04-16T04:55:24.995Z] 1. Register with subscription-manager [2026-04-16T04:55:24.995Z] # subscription-manager register [2026-04-16T04:55:24.995Z] [2026-04-16T04:55:24.995Z] 2. Register with insights-client [2026-04-16T04:55:24.995Z] # insights-client --register [2026-04-16T04:55:25.792Z] This system is currently not registered. [2026-04-16T04:55:28.274Z] Domain 'an-a01n02' destroyed [2026-04-16T04:55:28.274Z] [2026-04-16T04:55:28.648Z] Domain 'an-a01n02' has been undefined [2026-04-16T04:55:28.648Z] [2026-04-16T04:55:28.648Z] Warning: Permanently added 'an-a01dr01' (ED25519) to the list of known hosts. [2026-04-16T04:55:29.022Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-04-16T04:55:29.022Z] This might take a few seconds. [2026-04-16T04:55:29.022Z] [2026-04-16T04:55:29.022Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T04:55:33.114Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T04:55:33.114Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T04:55:33.114Z] [2026-04-16T04:55:33.114Z] Manage your connected systems: https://red.ht/connector [2026-04-16T04:55:33.114Z] [2026-04-16T04:55:33.114Z] The following errors were encountered during disconnect: [2026-04-16T04:55:33.114Z] [2026-04-16T04:55:33.114Z] TYPE STEP ERROR [2026-04-16T04:55:33.114Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T04:55:33.114Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T04:55:36.336Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T04:55:36.336Z] [2026-04-16T04:55:36.336Z] 1. Register with subscription-manager [2026-04-16T04:55:36.336Z] # subscription-manager register [2026-04-16T04:55:36.336Z] [2026-04-16T04:55:36.336Z] 2. Register with insights-client [2026-04-16T04:55:36.336Z] # insights-client --register [2026-04-16T04:55:37.135Z] This system is currently not registered. [2026-04-16T04:55:39.612Z] Domain 'an-a01dr01' destroyed [2026-04-16T04:55:39.612Z] [2026-04-16T04:55:39.612Z] Domain 'an-a01dr01' has been undefined [2026-04-16T04:55:39.612Z] [2026-04-16T04:55:39.612Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-16T04:55:39.985Z] Genering ssh keys and ssh config [2026-04-16T04:55:39.985Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:55:40.359Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:55:40.359Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:55:40.359Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:55:40.359Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:55:40.732Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-16T04:55:40.732Z] receiving incremental file list [2026-04-16T04:55:40.732Z] [2026-04-16T04:55:40.732Z] sent 27 bytes received 442 bytes 938.00 bytes/sec [2026-04-16T04:55:40.732Z] total size is 16,565,796,937 speedup is 35,321,528.65 [2026-04-16T04:55:40.732Z] Downloading latest AlmaLinux 9 cloud image [2026-04-16T04:55:41.108Z] Generating daily cache [2026-04-16T04:55:41.108Z] Updating daily cache for almalinux-9 2026-04-16 [2026-04-16T04:55:41.108Z] Destroying VMs (if any) [2026-04-16T04:55:41.108Z] Formatting 'almalinux-9-latest.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/images/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:55:41.108Z] wrote an-cache-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:55:42.946Z] [2026-04-16T04:55:42.946Z] Starting install... [2026-04-16T04:55:42.946Z] [2026-04-16T04:55:42.946Z] Creating domain... | 00:00 [2026-04-16T04:55:42.946Z] Domain creation completed. [2026-04-16T04:55:42.946Z] Waiting for nodes and cloud-init to complete full init [2026-04-16T04:55:46.169Z] cloud-init status node an-cache: [2026-04-16T04:55:46.169Z] Thu Apr 16 00:55:45 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:55:57.543Z] cloud-init status node an-cache: [2026-04-16T04:55:57.543Z] Thu Apr 16 00:55:56 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:56:08.907Z] cloud-init status node an-cache: running [2026-04-16T04:56:08.907Z] Thu Apr 16 00:56:07 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:56:18.361Z] cloud-init status node an-cache: running [2026-04-16T04:56:18.361Z] Thu Apr 16 00:56:18 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:56:29.719Z] cloud-init status node an-cache: running [2026-04-16T04:56:29.719Z] Thu Apr 16 00:56:28 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:56:41.109Z] cloud-init status node an-cache: running [2026-04-16T04:56:41.109Z] Thu Apr 16 00:56:39 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:56:50.506Z] cloud-init status node an-cache: running [2026-04-16T04:56:50.506Z] Thu Apr 16 00:56:50 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:57:01.864Z] cloud-init status node an-cache: running [2026-04-16T04:57:01.864Z] Thu Apr 16 00:57:01 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:57:13.236Z] cloud-init status node an-cache: running [2026-04-16T04:57:13.236Z] Thu Apr 16 00:57:12 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:57:24.603Z] cloud-init status node an-cache: running [2026-04-16T04:57:24.603Z] Thu Apr 16 00:57:23 EDT 2026 actual: 0 expected: 1 [2026-04-16T04:57:33.997Z] cloud-init status node an-cache: done [2026-04-16T04:57:33.997Z] Thu Apr 16 00:57:33 EDT 2026 actual: 1 expected: 1 [2026-04-16T04:59:29.614Z] bash: line 1: rhc: command not found [2026-04-16T04:59:29.614Z] bash: line 1: insights-client: command not found [2026-04-16T04:59:29.614Z] bash: line 1: subscription-manager: command not found [2026-04-16T04:59:29.614Z] Domain 'an-cache' is being shutdown [2026-04-16T04:59:29.614Z] [2026-04-16T04:59:32.093Z] Domain 'an-cache' has been undefined [2026-04-16T04:59:32.093Z] [2026-04-16T04:59:32.467Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:59:32.467Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:59:32.467Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:59:32.840Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:59:32.840Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T04:59:32.840Z] Done updating daily cache [2026-04-16T04:59:32.840Z] Creating nodes disks [2026-04-16T04:59:40.556Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T04:59:40.556Z] Generating an-test-deployX kickstart files/isos [2026-04-16T04:59:40.556Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-04-16T04:59:40.556Z] Media current: stdio file, overwriteable [2026-04-16T04:59:40.556Z] Media status : is blank [2026-04-16T04:59:40.556Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T04:59:40.556Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] ISO image produced: 184 sectors [2026-04-16T04:59:40.556Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T04:59:40.556Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-04-16T04:59:40.556Z] Media current: stdio file, overwriteable [2026-04-16T04:59:40.556Z] Media status : is blank [2026-04-16T04:59:40.556Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T04:59:40.556Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] ISO image produced: 184 sectors [2026-04-16T04:59:40.556Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T04:59:40.556Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-04-16T04:59:40.556Z] Media current: stdio file, overwriteable [2026-04-16T04:59:40.556Z] Media status : is blank [2026-04-16T04:59:40.556Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T04:59:40.556Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] ISO image produced: 184 sectors [2026-04-16T04:59:40.556Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T04:59:40.556Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-04-16T04:59:40.556Z] Media current: stdio file, overwriteable [2026-04-16T04:59:40.556Z] Media status : is blank [2026-04-16T04:59:40.556Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T04:59:40.556Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] ISO image produced: 184 sectors [2026-04-16T04:59:40.556Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T04:59:40.556Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T04:59:40.556Z] [2026-04-16T04:59:40.556Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-04-16T04:59:40.556Z] Media current: stdio file, overwriteable [2026-04-16T04:59:40.556Z] Media status : is blank [2026-04-16T04:59:40.556Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T04:59:40.556Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.556Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T04:59:40.557Z] ISO image produced: 184 sectors [2026-04-16T04:59:40.557Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T04:59:40.557Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-04-16T04:59:40.557Z] [2026-04-16T04:59:40.557Z] Generating cloud-init configs [2026-04-16T04:59:40.557Z] Generating cloud-init metadata and seed images [2026-04-16T04:59:40.557Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:59:40.557Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:59:40.557Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:59:40.929Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:59:40.929Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T04:59:40.929Z] Creating VMs [2026-04-16T04:59:42.768Z] [2026-04-16T04:59:42.768Z] Starting install... [2026-04-16T04:59:42.768Z] [2026-04-16T04:59:42.768Z] Creating domain... | 00:00 [2026-04-16T04:59:42.768Z] Domain creation completed. [2026-04-16T04:59:44.050Z] [2026-04-16T04:59:44.050Z] Starting install... [2026-04-16T04:59:44.050Z] [2026-04-16T04:59:44.050Z] Creating domain... | 00:00 [2026-04-16T04:59:44.050Z] Domain creation completed. [2026-04-16T04:59:45.896Z] [2026-04-16T04:59:45.896Z] Starting install... [2026-04-16T04:59:45.896Z] [2026-04-16T04:59:45.896Z] Creating domain... | 00:00 [2026-04-16T04:59:45.896Z] Domain creation completed. [2026-04-16T04:59:47.732Z] [2026-04-16T04:59:47.732Z] Starting install... [2026-04-16T04:59:47.732Z] [2026-04-16T04:59:47.732Z] Creating domain... | 00:00 [2026-04-16T04:59:47.732Z] Domain creation completed. [2026-04-16T04:59:49.016Z] [2026-04-16T04:59:49.016Z] Starting install... [2026-04-16T04:59:49.016Z] [2026-04-16T04:59:49.016Z] Creating domain... | 00:00 [2026-04-16T04:59:49.016Z] Domain creation completed. [2026-04-16T04:59:49.391Z] Waiting for nodes and cloud-init to complete full init [2026-04-16T04:59:52.617Z] cloud-init status node an-striker01: [2026-04-16T04:59:55.845Z] cloud-init status node an-striker02: [2026-04-16T04:59:59.071Z] cloud-init status node an-a01n01: [2026-04-16T05:00:01.553Z] cloud-init status node an-a01n02: [2026-04-16T05:00:04.780Z] cloud-init status node an-a01dr01: [2026-04-16T05:00:04.780Z] Thu Apr 16 01:00:04 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:00:16.151Z] cloud-init status node an-striker01: running [2026-04-16T05:00:17.996Z] cloud-init status node an-striker02: running [2026-04-16T05:00:18.796Z] cloud-init status node an-a01n01: running [2026-04-16T05:00:20.081Z] cloud-init status node an-a01n02: running [2026-04-16T05:00:21.365Z] cloud-init status node an-a01dr01: running [2026-04-16T05:00:21.365Z] Thu Apr 16 01:00:21 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:00:32.741Z] cloud-init status node an-striker01: running [2026-04-16T05:00:33.116Z] cloud-init status node an-striker02: running [2026-04-16T05:00:34.401Z] cloud-init status node an-a01n01: running [2026-04-16T05:00:35.201Z] cloud-init status node an-a01n02: running [2026-04-16T05:00:36.001Z] cloud-init status node an-a01dr01: running [2026-04-16T05:00:36.001Z] Thu Apr 16 01:00:35 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:00:47.374Z] cloud-init status node an-striker01: running [2026-04-16T05:00:47.749Z] cloud-init status node an-striker02: running [2026-04-16T05:00:49.033Z] cloud-init status node an-a01n01: running [2026-04-16T05:00:49.835Z] cloud-init status node an-a01n02: running [2026-04-16T05:00:50.634Z] cloud-init status node an-a01dr01: running [2026-04-16T05:00:50.634Z] Thu Apr 16 01:00:50 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:01:02.005Z] cloud-init status node an-striker01: running [2026-04-16T05:01:02.378Z] cloud-init status node an-striker02: running [2026-04-16T05:01:03.176Z] cloud-init status node an-a01n01: running [2026-04-16T05:01:04.461Z] cloud-init status node an-a01n02: running [2026-04-16T05:01:05.263Z] cloud-init status node an-a01dr01: running [2026-04-16T05:01:05.263Z] Thu Apr 16 01:01:04 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:01:16.629Z] cloud-init status node an-striker01: running [2026-04-16T05:01:17.005Z] cloud-init status node an-striker02: running [2026-04-16T05:01:17.803Z] cloud-init status node an-a01n01: running [2026-04-16T05:01:18.601Z] cloud-init status node an-a01n02: running [2026-04-16T05:01:19.885Z] cloud-init status node an-a01dr01: running [2026-04-16T05:01:19.885Z] Thu Apr 16 01:01:19 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:01:31.254Z] cloud-init status node an-striker01: running [2026-04-16T05:01:31.630Z] cloud-init status node an-striker02: running [2026-04-16T05:01:32.916Z] cloud-init status node an-a01n01: running [2026-04-16T05:01:33.715Z] cloud-init status node an-a01n02: running [2026-04-16T05:01:34.518Z] cloud-init status node an-a01dr01: running [2026-04-16T05:01:34.518Z] Thu Apr 16 01:01:34 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:01:45.927Z] cloud-init status node an-striker01: running [2026-04-16T05:01:46.728Z] cloud-init status node an-striker02: running [2026-04-16T05:01:47.530Z] cloud-init status node an-a01n01: running [2026-04-16T05:01:48.818Z] cloud-init status node an-a01n02: running [2026-04-16T05:01:49.620Z] cloud-init status node an-a01dr01: running [2026-04-16T05:01:49.620Z] Thu Apr 16 01:01:49 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:02:00.992Z] cloud-init status node an-striker01: running [2026-04-16T05:02:01.367Z] cloud-init status node an-striker02: running [2026-04-16T05:02:02.172Z] cloud-init status node an-a01n01: running [2026-04-16T05:02:02.970Z] cloud-init status node an-a01n02: running [2026-04-16T05:02:03.770Z] cloud-init status node an-a01dr01: running [2026-04-16T05:02:03.770Z] Thu Apr 16 01:02:03 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:02:15.717Z] cloud-init status node an-striker01: running [2026-04-16T05:02:15.717Z] cloud-init status node an-striker02: running [2026-04-16T05:02:16.516Z] cloud-init status node an-a01n01: running [2026-04-16T05:02:17.801Z] cloud-init status node an-a01n02: running [2026-04-16T05:02:18.600Z] cloud-init status node an-a01dr01: running [2026-04-16T05:02:18.600Z] Thu Apr 16 01:02:18 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:02:29.971Z] cloud-init status node an-striker01: running [2026-04-16T05:02:30.347Z] cloud-init status node an-striker02: running [2026-04-16T05:02:31.148Z] cloud-init status node an-a01n01: running [2026-04-16T05:02:31.946Z] cloud-init status node an-a01n02: running [2026-04-16T05:02:32.744Z] cloud-init status node an-a01dr01: running [2026-04-16T05:02:32.744Z] Thu Apr 16 01:02:32 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:02:44.115Z] cloud-init status node an-striker01: running [2026-04-16T05:02:44.913Z] cloud-init status node an-striker02: running [2026-04-16T05:02:45.712Z] cloud-init status node an-a01n01: running [2026-04-16T05:02:46.510Z] cloud-init status node an-a01n02: running [2026-04-16T05:02:47.311Z] cloud-init status node an-a01dr01: running [2026-04-16T05:02:47.311Z] Thu Apr 16 01:02:47 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:02:58.677Z] cloud-init status node an-striker01: running [2026-04-16T05:02:59.476Z] cloud-init status node an-striker02: running [2026-04-16T05:03:00.275Z] cloud-init status node an-a01n01: running [2026-04-16T05:03:01.072Z] cloud-init status node an-a01n02: running [2026-04-16T05:03:02.355Z] cloud-init status node an-a01dr01: running [2026-04-16T05:03:02.355Z] Thu Apr 16 01:03:01 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:03:13.719Z] cloud-init status node an-striker01: running [2026-04-16T05:03:14.093Z] cloud-init status node an-striker02: running [2026-04-16T05:03:14.892Z] cloud-init status node an-a01n01: running [2026-04-16T05:03:15.690Z] cloud-init status node an-a01n02: running [2026-04-16T05:03:16.974Z] cloud-init status node an-a01dr01: running [2026-04-16T05:03:16.974Z] Thu Apr 16 01:03:16 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:03:28.579Z] cloud-init status node an-striker01: running [2026-04-16T05:03:29.011Z] cloud-init status node an-striker02: running [2026-04-16T05:03:29.809Z] cloud-init status node an-a01n01: running [2026-04-16T05:03:31.092Z] cloud-init status node an-a01n02: running [2026-04-16T05:03:31.891Z] cloud-init status node an-a01dr01: running [2026-04-16T05:03:31.891Z] Thu Apr 16 01:03:31 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:03:43.260Z] cloud-init status node an-striker01: running [2026-04-16T05:03:43.639Z] cloud-init status node an-striker02: running [2026-04-16T05:03:44.442Z] cloud-init status node an-a01n01: running [2026-04-16T05:03:45.726Z] cloud-init status node an-a01n02: running [2026-04-16T05:03:46.524Z] cloud-init status node an-a01dr01: running [2026-04-16T05:03:46.524Z] Thu Apr 16 01:03:46 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:03:57.890Z] cloud-init status node an-striker01: running [2026-04-16T05:03:58.264Z] cloud-init status node an-striker02: running [2026-04-16T05:03:59.062Z] cloud-init status node an-a01n01: running [2026-04-16T05:04:00.344Z] cloud-init status node an-a01n02: running [2026-04-16T05:04:01.140Z] cloud-init status node an-a01dr01: running [2026-04-16T05:04:01.140Z] Thu Apr 16 01:04:00 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:04:12.648Z] cloud-init status node an-striker01: running [2026-04-16T05:04:13.023Z] cloud-init status node an-striker02: running [2026-04-16T05:04:13.821Z] cloud-init status node an-a01n01: running [2026-04-16T05:04:15.103Z] cloud-init status node an-a01n02: running [2026-04-16T05:04:15.902Z] cloud-init status node an-a01dr01: running [2026-04-16T05:04:15.902Z] Thu Apr 16 01:04:15 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:04:27.263Z] cloud-init status node an-striker01: running [2026-04-16T05:04:28.185Z] cloud-init status node an-striker02: running [2026-04-16T05:04:28.985Z] cloud-init status node an-a01n01: running [2026-04-16T05:04:29.784Z] cloud-init status node an-a01n02: running [2026-04-16T05:04:31.068Z] cloud-init status node an-a01dr01: running [2026-04-16T05:04:31.068Z] Thu Apr 16 01:04:30 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:04:42.431Z] cloud-init status node an-striker01: running [2026-04-16T05:04:42.809Z] cloud-init status node an-striker02: running [2026-04-16T05:04:43.610Z] cloud-init status node an-a01n01: running [2026-04-16T05:04:44.896Z] cloud-init status node an-a01n02: running [2026-04-16T05:04:45.695Z] cloud-init status node an-a01dr01: running [2026-04-16T05:04:45.695Z] Thu Apr 16 01:04:45 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:04:57.061Z] cloud-init status node an-striker01: running [2026-04-16T05:04:57.435Z] cloud-init status node an-striker02: running [2026-04-16T05:04:58.718Z] cloud-init status node an-a01n01: running [2026-04-16T05:04:59.518Z] cloud-init status node an-a01n02: running [2026-04-16T05:05:00.316Z] cloud-init status node an-a01dr01: running [2026-04-16T05:05:00.316Z] Thu Apr 16 01:05:00 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:05:11.676Z] cloud-init status node an-striker01: running [2026-04-16T05:05:12.054Z] cloud-init status node an-striker02: running [2026-04-16T05:05:13.335Z] cloud-init status node an-a01n01: running [2026-04-16T05:05:14.134Z] cloud-init status node an-a01n02: running [2026-04-16T05:05:14.933Z] cloud-init status node an-a01dr01: running [2026-04-16T05:05:14.933Z] Thu Apr 16 01:05:14 EDT 2026 actual: 0 expected: 5 [2026-04-16T05:05:26.430Z] cloud-init status node an-striker01: running [2026-04-16T05:05:26.804Z] cloud-init status node an-striker02: running [2026-04-16T05:05:28.090Z] cloud-init status node an-a01n01: running [2026-04-16T05:05:28.895Z] cloud-init status node an-a01n02: running [2026-04-16T05:05:29.699Z] cloud-init status node an-a01dr01: done [2026-04-16T05:05:29.699Z] Thu Apr 16 01:05:29 EDT 2026 actual: 1 expected: 5 [2026-04-16T05:05:41.114Z] cloud-init status node an-striker01: running [2026-04-16T05:05:42.975Z] cloud-init status node an-striker02: running [2026-04-16T05:05:43.782Z] cloud-init status node an-a01n01: done [2026-04-16T05:05:45.079Z] cloud-init status node an-a01n02: running [2026-04-16T05:05:46.374Z] cloud-init status node an-a01dr01: done [2026-04-16T05:05:46.374Z] Thu Apr 16 01:05:46 EDT 2026 actual: 2 expected: 5 [2026-04-16T05:06:00.206Z] cloud-init status node an-striker01: running [2026-04-16T05:06:00.206Z] cloud-init status node an-striker02: running [2026-04-16T05:06:01.491Z] cloud-init status node an-a01n01: done [2026-04-16T05:06:02.775Z] cloud-init status node an-a01n02: done [2026-04-16T05:06:03.574Z] cloud-init status node an-a01dr01: done [2026-04-16T05:06:03.574Z] Thu Apr 16 01:06:03 EDT 2026 actual: 3 expected: 5 [2026-04-16T05:06:17.285Z] cloud-init status node an-striker01: running [2026-04-16T05:06:17.285Z] cloud-init status node an-striker02: running [2026-04-16T05:06:17.659Z] cloud-init status node an-a01n01: done [2026-04-16T05:06:18.941Z] cloud-init status node an-a01n02: done [2026-04-16T05:06:19.740Z] cloud-init status node an-a01dr01: done [2026-04-16T05:06:19.740Z] Thu Apr 16 01:06:19 EDT 2026 actual: 3 expected: 5 [2026-04-16T05:06:33.448Z] cloud-init status node an-striker01: running [2026-04-16T05:06:33.448Z] cloud-init status node an-striker02: running [2026-04-16T05:06:33.448Z] cloud-init status node an-a01n01: done [2026-04-16T05:06:34.732Z] cloud-init status node an-a01n02: done [2026-04-16T05:06:35.531Z] cloud-init status node an-a01dr01: done [2026-04-16T05:06:35.531Z] Thu Apr 16 01:06:35 EDT 2026 actual: 3 expected: 5 [2026-04-16T05:06:47.402Z] cloud-init status node an-striker01: done [2026-04-16T05:06:47.776Z] cloud-init status node an-striker02: done [2026-04-16T05:06:49.058Z] cloud-init status node an-a01n01: done [2026-04-16T05:06:49.857Z] cloud-init status node an-a01n02: done [2026-04-16T05:06:50.655Z] cloud-init status node an-a01dr01: done [2026-04-16T05:06:50.656Z] Thu Apr 16 01:06:50 EDT 2026 actual: 5 expected: 5 [2026-04-16T05:06:50.656Z] Building simengine model [2026-04-16T05:07:04.361Z] Generating anvil striker auto config [2026-04-16T05:07:07.588Z] Cloning into 'anvil-test-suite'... [2026-04-16T05:07:07.962Z] generated ansible config: [2026-04-16T05:07:07.962Z] ========================= [2026-04-16T05:07:07.962Z] [defaults] [2026-04-16T05:07:07.962Z] inventory = ansible-inventory [2026-04-16T05:07:07.962Z] forks = 500 [2026-04-16T05:07:07.962Z] any_errors_fatal = true [2026-04-16T05:07:07.962Z] display_args_to_stdout = true [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:07.962Z] [ssh_connection] [2026-04-16T05:07:07.962Z] retries = 20 [2026-04-16T05:07:07.962Z] ========================= [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:07.962Z] generated ansible inventory: [2026-04-16T05:07:07.962Z] ============================ [2026-04-16T05:07:07.962Z] [strikers] [2026-04-16T05:07:07.962Z] an-striker01 [2026-04-16T05:07:07.962Z] an-striker02 [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:07.962Z] [nodes] [2026-04-16T05:07:07.962Z] an-a01n01 [2026-04-16T05:07:07.962Z] an-a01n02 [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:07.962Z] [drnodes] [2026-04-16T05:07:07.962Z] an-a01dr01 [2026-04-16T05:07:07.962Z] ============================ [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:07.962Z] Checking connectivity to nodes [2026-04-16T05:07:07.962Z] [2026-04-16T05:07:10.241Z] an-a01dr01 | SUCCESS => { [2026-04-16T05:07:10.241Z] "ansible_facts": { [2026-04-16T05:07:10.241Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T05:07:10.241Z] }, [2026-04-16T05:07:10.241Z] "changed": false, [2026-04-16T05:07:10.241Z] "ping": "pong" [2026-04-16T05:07:10.241Z] } [2026-04-16T05:07:10.241Z] an-a01n01 | SUCCESS => { [2026-04-16T05:07:10.241Z] "ansible_facts": { [2026-04-16T05:07:10.241Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T05:07:10.241Z] }, [2026-04-16T05:07:10.241Z] "changed": false, [2026-04-16T05:07:10.241Z] "ping": "pong" [2026-04-16T05:07:10.241Z] } [2026-04-16T05:07:10.241Z] an-a01n02 | SUCCESS => { [2026-04-16T05:07:10.241Z] "ansible_facts": { [2026-04-16T05:07:10.241Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T05:07:10.241Z] }, [2026-04-16T05:07:10.241Z] "changed": false, [2026-04-16T05:07:10.241Z] "ping": "pong" [2026-04-16T05:07:10.241Z] } [2026-04-16T05:07:10.241Z] an-striker01 | SUCCESS => { [2026-04-16T05:07:10.241Z] "ansible_facts": { [2026-04-16T05:07:10.241Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T05:07:10.241Z] }, [2026-04-16T05:07:10.241Z] "changed": false, [2026-04-16T05:07:10.241Z] "ping": "pong" [2026-04-16T05:07:10.241Z] } [2026-04-16T05:07:10.241Z] an-striker02 | SUCCESS => { [2026-04-16T05:07:10.241Z] "ansible_facts": { [2026-04-16T05:07:10.241Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T05:07:10.241Z] }, [2026-04-16T05:07:10.241Z] "changed": false, [2026-04-16T05:07:10.241Z] "ping": "pong" [2026-04-16T05:07:10.241Z] } [2026-04-16T05:07:10.241Z] [2026-04-16T05:07:11.044Z] [2026-04-16T05:07:11.044Z] PLAY [Setup all nodes] ********************************************************* [2026-04-16T05:07:11.044Z] [2026-04-16T05:07:11.044Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:07:13.532Z] ok: [an-a01dr01] [2026-04-16T05:07:13.532Z] ok: [an-a01n01] [2026-04-16T05:07:13.532Z] ok: [an-a01n02] [2026-04-16T05:07:13.532Z] ok: [an-striker01] [2026-04-16T05:07:13.532Z] ok: [an-striker02] [2026-04-16T05:07:13.532Z] [2026-04-16T05:07:13.532Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-04-16T05:07:14.821Z] changed: [an-a01dr01] [2026-04-16T05:07:14.821Z] changed: [an-a01n02] [2026-04-16T05:07:14.821Z] changed: [an-striker01] [2026-04-16T05:07:14.821Z] changed: [an-a01n01] [2026-04-16T05:07:14.821Z] changed: [an-striker02] [2026-04-16T05:07:14.821Z] [2026-04-16T05:07:14.821Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-04-16T05:07:15.622Z] ok: [an-a01dr01] [2026-04-16T05:07:15.622Z] ok: [an-a01n02] [2026-04-16T05:07:15.622Z] ok: [an-striker01] [2026-04-16T05:07:15.622Z] ok: [an-a01n01] [2026-04-16T05:07:15.622Z] ok: [an-striker02] [2026-04-16T05:07:15.622Z] [2026-04-16T05:07:15.622Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-04-16T05:07:17.462Z] changed: [an-a01dr01] [2026-04-16T05:07:17.462Z] changed: [an-a01n01] [2026-04-16T05:07:17.462Z] changed: [an-a01n02] [2026-04-16T05:07:17.462Z] changed: [an-striker01] [2026-04-16T05:07:17.462Z] changed: [an-striker02] [2026-04-16T05:07:17.462Z] [2026-04-16T05:07:17.462Z] PLAY [setup strikers] ********************************************************** [2026-04-16T05:07:17.462Z] [2026-04-16T05:07:17.462Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:07:19.308Z] ok: [an-striker01] [2026-04-16T05:07:19.308Z] ok: [an-striker02] [2026-04-16T05:07:19.308Z] [2026-04-16T05:07:19.308Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-04-16T05:07:21.150Z] changed: [an-striker01] [2026-04-16T05:07:21.150Z] changed: [an-striker02] [2026-04-16T05:07:21.150Z] [2026-04-16T05:07:21.150Z] PLAY [setup lvm storage] ******************************************************* [2026-04-16T05:07:21.150Z] [2026-04-16T05:07:21.150Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:07:22.439Z] ok: [an-a01n02] [2026-04-16T05:07:22.439Z] ok: [an-a01dr01] [2026-04-16T05:07:22.439Z] ok: [an-a01n01] [2026-04-16T05:07:22.439Z] [2026-04-16T05:07:22.439Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-04-16T05:07:22.439Z] ] **************** [2026-04-16T05:07:23.239Z] ok: [an-a01n02] [2026-04-16T05:07:23.239Z] ok: [an-a01n01] [2026-04-16T05:07:23.239Z] ok: [an-a01dr01] [2026-04-16T05:07:23.239Z] [2026-04-16T05:07:23.239Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-04-16T05:07:23.239Z] skipping: [an-a01n01] [2026-04-16T05:07:23.239Z] [2026-04-16T05:07:23.239Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-04-16T05:07:24.527Z] changed: [an-a01dr01] [2026-04-16T05:07:24.527Z] changed: [an-a01n02] [2026-04-16T05:07:24.527Z] changed: [an-a01n01] [2026-04-16T05:07:24.527Z] [2026-04-16T05:07:24.527Z] PLAY [deploy anvil] ************************************************************ [2026-04-16T05:07:24.527Z] [2026-04-16T05:07:24.527Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:07:26.475Z] ok: [an-striker01] [2026-04-16T05:07:26.475Z] ok: [an-striker02] [2026-04-16T05:07:26.475Z] [2026-04-16T05:07:26.475Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-04-16T05:07:26.854Z] ok: [an-striker02] [2026-04-16T05:07:26.854Z] ok: [an-striker01] [2026-04-16T05:07:26.854Z] [2026-04-16T05:07:26.854Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-04-16T05:07:28.705Z] changed: [an-striker02] [2026-04-16T05:07:28.705Z] changed: [an-striker01] [2026-04-16T05:07:28.705Z] [2026-04-16T05:07:28.705Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-04-16T05:07:28.705Z] ] *** [2026-04-16T05:07:31.203Z] changed: [an-striker01] [2026-04-16T05:07:31.203Z] changed: [an-striker02] [2026-04-16T05:07:31.203Z] [2026-04-16T05:07:31.203Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:07:31.203Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:07:31.203Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-04-16T05:07:31.203Z] ] *** [2026-04-16T05:13:12.749Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-16T05:13:12.749Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-16T05:13:12.749Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-16T05:13:12.749Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-04-16T05:13:12.750Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-04-16T05:14:51.964Z] ok: [an-striker02 -> localhost] [2026-04-16T05:14:51.964Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-04-16T05:14:51.964Z] ok: [an-striker01 -> localhost] [2026-04-16T05:14:51.964Z] [2026-04-16T05:14:51.964Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-04-16T05:14:51.964Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-04-16T05:14:51.964Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-04-16T05:14:51.964Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-04-16T05:14:51.964Z] echo "DB has completed sync" [2026-04-16T05:14:51.964Z] exit 0 [2026-04-16T05:14:51.964Z] fi [2026-04-16T05:14:51.964Z] if [ "$dbcon" != 2 ]; then [2026-04-16T05:14:51.964Z] echo "Detected $dbcon db connections (expected 2)" [2026-04-16T05:14:51.964Z] exit 1 [2026-04-16T05:14:51.964Z] fi [2026-04-16T05:14:51.964Z] ] *** [2026-04-16T05:15:32.878Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-04-16T05:15:32.878Z] ok: [an-striker01] [2026-04-16T05:15:32.878Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-04-16T05:15:32.878Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left). [2026-04-16T05:15:32.878Z] ok: [an-striker02] [2026-04-16T05:15:32.878Z] [2026-04-16T05:15:32.878Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:15:32.878Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:15:32.878Z] --username=postgres anvil -t \ [2026-04-16T05:15:32.878Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-04-16T05:15:32.878Z] grep altee | \ [2026-04-16T05:15:32.878Z] grep 1$ | \ [2026-04-16T05:15:32.878Z] wc -l [2026-04-16T05:15:32.878Z] ] *** [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-16T05:21:14.116Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-16T05:21:14.117Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-16T05:21:14.117Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-16T05:21:14.117Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-16T05:21:14.117Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-16T05:23:54.102Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-16T05:23:54.102Z] ok: [an-striker01 -> localhost] [2026-04-16T05:23:54.102Z] ok: [an-striker02 -> localhost] [2026-04-16T05:23:54.102Z] [2026-04-16T05:23:54.102Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:23:54.102Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:23:54.102Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-04-16T05:23:54.102Z] grep altee | \ [2026-04-16T05:23:54.102Z] wc -l [2026-04-16T05:23:54.102Z] ] *** [2026-04-16T05:24:03.494Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-04-16T05:24:03.494Z] ok: [an-striker02 -> localhost] [2026-04-16T05:24:03.494Z] ok: [an-striker01 -> localhost] [2026-04-16T05:24:03.494Z] [2026-04-16T05:24:03.494Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:24:03.494Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:24:03.494Z] --username=postgres anvil -t \ [2026-04-16T05:24:03.494Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-04-16T05:24:03.494Z] grep altee | \ [2026-04-16T05:24:03.494Z] grep 1$ | \ [2026-04-16T05:24:03.494Z] wc -l [2026-04-16T05:24:03.494Z] ] *** [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (346 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (346 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (345 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (345 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (344 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (344 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (343 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (343 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (342 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (342 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (341 retries left). [2026-04-16T05:27:28.845Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (341 retries left). [2026-04-16T05:27:28.845Z] ok: [an-striker02 -> localhost] [2026-04-16T05:27:28.845Z] ok: [an-striker01 -> localhost] [2026-04-16T05:27:28.845Z] [2026-04-16T05:27:28.845Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:27:28.845Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:27:28.845Z] --username=postgres anvil -t \ [2026-04-16T05:27:28.845Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-04-16T05:27:28.845Z] grep altee | \ [2026-04-16T05:27:28.845Z] grep 1$ | \ [2026-04-16T05:27:28.845Z] wc -l [2026-04-16T05:27:28.845Z] ] *** [2026-04-16T05:27:52.863Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-16T05:27:52.863Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-16T05:27:52.863Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-16T05:27:52.863Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-16T05:27:52.863Z] ok: [an-striker01 -> localhost] [2026-04-16T05:27:52.863Z] ok: [an-striker02 -> localhost] [2026-04-16T05:27:52.863Z] [2026-04-16T05:27:52.863Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T05:27:52.863Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T05:27:52.863Z] --username=postgres anvil -t \ [2026-04-16T05:27:52.863Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-04-16T05:27:52.863Z] grep altee | \ [2026-04-16T05:27:52.863Z] grep 1$ | \ [2026-04-16T05:27:52.863Z] wc -l [2026-04-16T05:27:52.863Z] ] *** [2026-04-16T05:27:52.863Z] ok: [an-striker01 -> localhost] [2026-04-16T05:27:52.863Z] ok: [an-striker02 -> localhost] [2026-04-16T05:27:52.863Z] [2026-04-16T05:27:52.863Z] PLAY [check network status after deployment] *********************************** [2026-04-16T05:27:52.863Z] [2026-04-16T05:27:52.863Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:27:52.863Z] ok: [an-striker01] [2026-04-16T05:27:52.863Z] ok: [an-a01dr01] [2026-04-16T05:27:52.863Z] ok: [an-a01n02] [2026-04-16T05:27:52.863Z] ok: [an-a01n01] [2026-04-16T05:27:52.863Z] ok: [an-striker02] [2026-04-16T05:27:52.863Z] [2026-04-16T05:27:52.863Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-04-16T05:27:52.863Z] if [ "$device" = "" ]; then [2026-04-16T05:27:52.863Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-04-16T05:27:52.863Z] exit 1 [2026-04-16T05:27:52.863Z] fi [2026-04-16T05:27:52.863Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-04-16T05:27:52.863Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-04-16T05:27:52.863Z] exit 1 [2026-04-16T05:27:52.863Z] fi [2026-04-16T05:27:52.863Z] exit 0 [2026-04-16T05:27:52.863Z] ] *** [2026-04-16T05:28:00.596Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=sn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=bcn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=ifn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=ifn1_bridge1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=ifn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=bcn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=sn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=sn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=bcn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=ifn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-04-16T05:28:00.597Z] skipping: [an-a01n02] => (item=lo) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-04-16T05:28:00.597Z] skipping: [an-a01n01] => (item=lo) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=bcn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=bcn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=sn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=sn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=ifn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=sn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-04-16T05:28:00.597Z] changed: [an-striker01] => (item=ifn1_link2) [2026-04-16T05:28:00.597Z] skipping: [an-striker01] => (item=lo) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=bcn1_bridge1) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=bcn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=sn1_link1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=ifn1_bond1) [2026-04-16T05:28:00.597Z] skipping: [an-a01dr01] => (item=lo) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-striker02] => (item=ifn1_link1) [2026-04-16T05:28:00.597Z] skipping: [an-striker02] => (item=lo) [2026-04-16T05:28:00.597Z] changed: [an-a01dr01] => (item=bcn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-04-16T05:28:00.597Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-04-16T05:28:00.597Z] [2026-04-16T05:28:00.597Z] PLAY [check pacemaker status after deployment] ********************************* [2026-04-16T05:28:00.597Z] [2026-04-16T05:28:00.597Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:28:01.880Z] ok: [an-a01n01] [2026-04-16T05:28:01.880Z] ok: [an-a01n02] [2026-04-16T05:28:01.880Z] [2026-04-16T05:28:01.880Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-04-16T05:28:01.880Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T05:28:01.880Z] [2026-04-16T05:28:01.880Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T05:28:01.880Z] ] *** [2026-04-16T05:28:05.976Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T05:28:05.976Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T05:28:05.976Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T05:28:05.976Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T05:28:05.976Z] [2026-04-16T05:28:05.976Z] PLAY [check drbd kmod] ********************************************************* [2026-04-16T05:28:05.976Z] [2026-04-16T05:28:05.976Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:28:07.258Z] ok: [an-a01n02] [2026-04-16T05:28:07.258Z] ok: [an-a01n01] [2026-04-16T05:28:07.258Z] ok: [an-a01dr01] [2026-04-16T05:28:07.258Z] [2026-04-16T05:28:07.258Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-04-16T05:28:07.258Z] ] *** [2026-04-16T05:28:08.060Z] ok: [an-a01n02] [2026-04-16T05:28:08.060Z] ok: [an-a01n01] [2026-04-16T05:28:08.060Z] ok: [an-a01dr01] [2026-04-16T05:28:08.060Z] [2026-04-16T05:28:08.060Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-04-16T05:28:08.060Z] ] *********************** [2026-04-16T05:28:08.060Z] skipping: [an-a01n01] [2026-04-16T05:28:08.060Z] skipping: [an-a01n02] [2026-04-16T05:28:08.060Z] skipping: [an-a01dr01] [2026-04-16T05:28:08.060Z] [2026-04-16T05:28:08.060Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-04-16T05:28:08.060Z] ] *** [2026-04-16T05:28:08.060Z] skipping: [an-a01n01] [2026-04-16T05:28:08.060Z] skipping: [an-a01n02] [2026-04-16T05:28:08.060Z] skipping: [an-a01dr01] [2026-04-16T05:28:08.060Z] [2026-04-16T05:28:08.060Z] PLAY [Copy iso images to striker01] ******************************************** [2026-04-16T05:28:08.060Z] [2026-04-16T05:28:08.060Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:28:09.898Z] ok: [an-striker01] [2026-04-16T05:28:09.898Z] [2026-04-16T05:28:09.898Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-04-16T05:28:10.696Z] ok: [an-striker01] [2026-04-16T05:28:10.696Z] [2026-04-16T05:28:10.696Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-04-16T05:28:11.070Z] changed: [an-striker01] [2026-04-16T05:28:11.070Z] [2026-04-16T05:28:11.070Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-04-16T05:28:18.789Z] changed: [an-striker01] => (item=deploy1.iso) [2026-04-16T05:28:18.789Z] changed: [an-striker01] => (item=deploy2.iso) [2026-04-16T05:28:18.789Z] changed: [an-striker01] => (item=deploy3.iso) [2026-04-16T05:28:18.789Z] changed: [an-striker01] => (item=deploy4.iso) [2026-04-16T05:28:18.789Z] changed: [an-striker01] => (item=deploy5.iso) [2026-04-16T05:28:18.789Z] [2026-04-16T05:28:18.789Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-04-16T05:28:18.789Z] ] *** [2026-04-16T05:28:19.587Z] changed: [an-striker01] [2026-04-16T05:28:19.587Z] [2026-04-16T05:28:19.587Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-04-16T05:28:20.870Z] changed: [an-striker01] [2026-04-16T05:28:20.870Z] [2026-04-16T05:28:20.870Z] PLAY [Check iso images propagation] ******************************************** [2026-04-16T05:28:20.870Z] [2026-04-16T05:28:20.870Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:28:23.352Z] ok: [an-a01n02] [2026-04-16T05:28:23.352Z] ok: [an-a01n01] [2026-04-16T05:28:23.352Z] ok: [an-a01dr01] [2026-04-16T05:28:23.352Z] ok: [an-striker01] [2026-04-16T05:28:23.352Z] ok: [an-striker02] [2026-04-16T05:28:23.352Z] [2026-04-16T05:28:23.352Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-04-16T05:28:23.352Z] exit 1 [2026-04-16T05:28:23.352Z] else [2026-04-16T05:28:23.352Z] exit 0 [2026-04-16T05:28:23.352Z] fi [2026-04-16T05:28:23.352Z] ] *** [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T05:30:02.581Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] ok: [an-striker02] [2026-04-16T05:30:02.581Z] ok: [an-a01n02] [2026-04-16T05:30:02.581Z] ok: [an-a01dr01] [2026-04-16T05:30:02.581Z] ok: [an-a01n01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] PLAY [example tests] *********************************************************** [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:30:02.581Z] ok: [an-a01dr01] [2026-04-16T05:30:02.581Z] ok: [an-a01n02] [2026-04-16T05:30:02.581Z] ok: [an-a01n01] [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] ok: [an-striker02] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-04-16T05:30:02.581Z] ] *** [2026-04-16T05:30:02.581Z] skipping: [an-a01n01] [2026-04-16T05:30:02.581Z] skipping: [an-a01n02] [2026-04-16T05:30:02.581Z] skipping: [an-a01dr01] [2026-04-16T05:30:02.581Z] ok: [an-striker02] [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-04-16T05:30:02.581Z] ] *** [2026-04-16T05:30:02.581Z] skipping: [an-striker01] [2026-04-16T05:30:02.581Z] skipping: [an-striker02] [2026-04-16T05:30:02.581Z] ok: [an-a01dr01] [2026-04-16T05:30:02.581Z] ok: [an-a01n01] [2026-04-16T05:30:02.581Z] ok: [an-a01n02] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [inject failure _raw_params=/usr/bin/false [2026-04-16T05:30:02.581Z] ] ******************************* [2026-04-16T05:30:02.581Z] skipping: [an-striker01] [2026-04-16T05:30:02.581Z] skipping: [an-striker02] [2026-04-16T05:30:02.581Z] skipping: [an-a01n01] [2026-04-16T05:30:02.581Z] skipping: [an-a01n02] [2026-04-16T05:30:02.581Z] skipping: [an-a01dr01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] PLAY [Record anvil available resources from striker01] ************************* [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-04-16T05:30:02.581Z] ] *** [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] PLAY [check selinux status] **************************************************** [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:30:02.581Z] ok: [an-a01n01] [2026-04-16T05:30:02.581Z] ok: [an-a01n02] [2026-04-16T05:30:02.581Z] ok: [an-a01dr01] [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] ok: [an-striker02] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-04-16T05:30:02.581Z] skipping: [an-striker01] [2026-04-16T05:30:02.581Z] skipping: [an-striker02] [2026-04-16T05:30:02.581Z] skipping: [an-a01n01] [2026-04-16T05:30:02.581Z] skipping: [an-a01n02] [2026-04-16T05:30:02.581Z] skipping: [an-a01dr01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-04-16T05:30:02.581Z] ok: [an-striker01] => { [2026-04-16T05:30:02.581Z] "ansible_selinux": { [2026-04-16T05:30:02.581Z] "config_mode": "enforcing", [2026-04-16T05:30:02.581Z] "mode": "enforcing", [2026-04-16T05:30:02.581Z] "policyvers": 33, [2026-04-16T05:30:02.581Z] "status": "enabled", [2026-04-16T05:30:02.581Z] "type": "targeted" [2026-04-16T05:30:02.581Z] }, [2026-04-16T05:30:02.581Z] "failed_when_result": false [2026-04-16T05:30:02.581Z] } [2026-04-16T05:30:02.581Z] ok: [an-striker02] => { [2026-04-16T05:30:02.581Z] "ansible_selinux": { [2026-04-16T05:30:02.581Z] "config_mode": "enforcing", [2026-04-16T05:30:02.581Z] "mode": "enforcing", [2026-04-16T05:30:02.581Z] "policyvers": 33, [2026-04-16T05:30:02.581Z] "status": "enabled", [2026-04-16T05:30:02.581Z] "type": "targeted" [2026-04-16T05:30:02.581Z] }, [2026-04-16T05:30:02.581Z] "failed_when_result": false [2026-04-16T05:30:02.581Z] } [2026-04-16T05:30:02.581Z] ok: [an-a01n01] => { [2026-04-16T05:30:02.581Z] "ansible_selinux": { [2026-04-16T05:30:02.581Z] "config_mode": "enforcing", [2026-04-16T05:30:02.581Z] "mode": "enforcing", [2026-04-16T05:30:02.581Z] "policyvers": 33, [2026-04-16T05:30:02.581Z] "status": "enabled", [2026-04-16T05:30:02.581Z] "type": "targeted" [2026-04-16T05:30:02.581Z] }, [2026-04-16T05:30:02.581Z] "failed_when_result": false [2026-04-16T05:30:02.581Z] } [2026-04-16T05:30:02.581Z] ok: [an-a01n02] => { [2026-04-16T05:30:02.581Z] "ansible_selinux": { [2026-04-16T05:30:02.581Z] "config_mode": "enforcing", [2026-04-16T05:30:02.581Z] "mode": "enforcing", [2026-04-16T05:30:02.581Z] "policyvers": 33, [2026-04-16T05:30:02.581Z] "status": "enabled", [2026-04-16T05:30:02.581Z] "type": "targeted" [2026-04-16T05:30:02.581Z] }, [2026-04-16T05:30:02.581Z] "failed_when_result": false [2026-04-16T05:30:02.581Z] } [2026-04-16T05:30:02.581Z] ok: [an-a01dr01] => { [2026-04-16T05:30:02.581Z] "ansible_selinux": { [2026-04-16T05:30:02.581Z] "config_mode": "enforcing", [2026-04-16T05:30:02.581Z] "mode": "enforcing", [2026-04-16T05:30:02.581Z] "policyvers": 33, [2026-04-16T05:30:02.581Z] "status": "enabled", [2026-04-16T05:30:02.581Z] "type": "targeted" [2026-04-16T05:30:02.581Z] }, [2026-04-16T05:30:02.581Z] "failed_when_result": false [2026-04-16T05:30:02.581Z] } [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] PLAY [Create random file on striker01] ***************************************** [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:30:02.581Z] ok: [an-striker01] [2026-04-16T05:30:02.581Z] [2026-04-16T05:30:02.581Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-04-16T05:30:05.062Z] changed: [an-striker01] [2026-04-16T05:30:05.062Z] [2026-04-16T05:30:05.062Z] PLAY [Check archive file propagation] ****************************************** [2026-04-16T05:30:05.062Z] [2026-04-16T05:30:05.062Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:30:06.901Z] ok: [an-a01n02] [2026-04-16T05:30:06.901Z] ok: [an-a01n01] [2026-04-16T05:30:06.901Z] ok: [an-a01dr01] [2026-04-16T05:30:06.901Z] ok: [an-striker02] [2026-04-16T05:30:06.901Z] ok: [an-striker01] [2026-04-16T05:30:06.901Z] [2026-04-16T05:30:06.901Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-04-16T05:30:06.901Z] exit 1 [2026-04-16T05:30:06.901Z] else [2026-04-16T05:30:06.901Z] exit 0 [2026-04-16T05:30:06.901Z] fi [2026-04-16T05:30:06.901Z] ] *** [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T05:31:31.230Z] ok: [an-striker01] [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T05:31:31.230Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T05:31:31.230Z] ok: [an-striker02] [2026-04-16T05:31:31.230Z] ok: [an-a01dr01] [2026-04-16T05:31:31.230Z] ok: [an-a01n02] [2026-04-16T05:31:31.230Z] ok: [an-a01n01] [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] PLAY [test fencing] ************************************************************ [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:31:31.230Z] ok: [an-a01n01] [2026-04-16T05:31:31.230Z] ok: [an-a01n02] [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T05:31:31.230Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T05:31:31.230Z] ] *** [2026-04-16T05:31:31.230Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T05:31:31.230Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T05:31:31.230Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T05:31:31.230Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-04-16T05:31:31.230Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T05:31:31.230Z] ] ********** [2026-04-16T05:31:31.230Z] skipping: [an-a01n02] [2026-04-16T05:31:31.230Z] changed: [an-a01n01] [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T05:31:31.230Z] skipping: [an-a01n02] [2026-04-16T05:31:31.230Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-04-16T05:31:31.230Z] [2026-04-16T05:31:31.230Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T05:34:41.226Z] ok: [an-a01n01 -> localhost] [2026-04-16T05:34:41.226Z] [2026-04-16T05:34:41.226Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T05:34:41.226Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T05:34:41.226Z] [2026-04-16T05:34:41.226Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T05:34:41.226Z] ] *** [2026-04-16T05:34:43.087Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T05:34:43.087Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T05:34:43.087Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T05:34:43.087Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T05:34:43.087Z] [2026-04-16T05:34:43.087Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-04-16T05:34:43.087Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T05:34:43.087Z] [2026-04-16T05:34:43.087Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T05:34:43.087Z] ] ********** [2026-04-16T05:34:43.462Z] skipping: [an-a01n01] [2026-04-16T05:34:43.462Z] changed: [an-a01n02] [2026-04-16T05:34:43.462Z] [2026-04-16T05:34:43.462Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T05:34:43.836Z] skipping: [an-a01n01] [2026-04-16T05:34:43.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-04-16T05:34:43.836Z] [2026-04-16T05:34:43.836Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T05:37:54.043Z] ok: [an-a01n02 -> localhost] [2026-04-16T05:37:54.043Z] [2026-04-16T05:37:54.043Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T05:37:54.417Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T05:37:54.417Z] [2026-04-16T05:37:54.417Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T05:37:54.417Z] ] *** [2026-04-16T05:37:58.512Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T05:37:58.512Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T05:37:58.512Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T05:37:58.512Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T05:37:58.512Z] [2026-04-16T05:37:58.512Z] PLAY [advanced server tests] *************************************************** [2026-04-16T05:37:58.512Z] [2026-04-16T05:37:58.512Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T05:38:00.350Z] ok: [an-a01n01] [2026-04-16T05:38:00.350Z] ok: [an-a01n02] [2026-04-16T05:38:00.350Z] [2026-04-16T05:38:00.350Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T05:38:00.350Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T05:38:00.350Z] [2026-04-16T05:38:00.350Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T05:38:00.350Z] ] *** [2026-04-16T05:38:04.448Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T05:38:04.448Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T05:38:04.448Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T05:38:04.448Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T05:38:04.448Z] [2026-04-16T05:38:04.448Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T05:38:04.448Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T05:38:04.448Z] [2026-04-16T05:38:04.448Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T05:38:04.448Z] if [ -z "$storage" ]; then [2026-04-16T05:38:04.448Z] echo "Unable to detect shared storage" [2026-04-16T05:38:04.448Z] exit 1 [2026-04-16T05:38:04.448Z] fi [2026-04-16T05:38:04.448Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T05:38:04.448Z] ] *** [2026-04-16T05:38:10.918Z] changed: [an-a01n01] [2026-04-16T05:38:10.918Z] changed: [an-a01n02] [2026-04-16T05:38:10.918Z] [2026-04-16T05:38:10.918Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T05:38:10.918Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T05:38:10.918Z] [2026-04-16T05:38:10.918Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T05:38:10.918Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T05:38:10.918Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T05:38:10.918Z] echo "Unable to detect all deploy iso" [2026-04-16T05:38:10.918Z] echo "Detected isos" [2026-04-16T05:38:10.918Z] echo "$isos" [2026-04-16T05:38:10.918Z] exit 1 [2026-04-16T05:38:10.918Z] fi [2026-04-16T05:38:10.918Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T05:38:10.918Z] if [ -z "$installiso" ]; then [2026-04-16T05:38:10.918Z] echo "Unable to detect install iso" [2026-04-16T05:38:10.918Z] echo "Detected isos" [2026-04-16T05:38:10.918Z] echo "$isos" [2026-04-16T05:38:10.918Z] exit 1 [2026-04-16T05:38:10.918Z] fi [2026-04-16T05:38:10.918Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T05:38:10.918Z] ] *** [2026-04-16T05:38:15.017Z] changed: [an-a01n01] [2026-04-16T05:38:15.017Z] changed: [an-a01n02] [2026-04-16T05:38:15.017Z] [2026-04-16T05:38:15.017Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-04-16T05:38:15.391Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T05:38:15.391Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T05:38:15.391Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T05:38:15.391Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T05:38:15.391Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T05:38:15.391Z] [2026-04-16T05:38:15.391Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T05:38:15.391Z] ] *** [2026-04-16T05:38:16.673Z] skipping: [an-a01n02] [2026-04-16T05:38:16.673Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.877934", "end": "2026-04-16 05:38:16.516668", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 05:38:15.638734", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T05:38:16.673Z] ...ignoring [2026-04-16T05:38:16.673Z] [2026-04-16T05:38:16.673Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T05:38:16.673Z] --ci-test \ [2026-04-16T05:38:16.673Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T05:38:16.673Z] --os almalinux9 \ [2026-04-16T05:38:16.673Z] --cpu 2 --ram 4G \ [2026-04-16T05:38:16.673Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T05:38:16.673Z] --install-media {{ installisouuid }} \ [2026-04-16T05:38:16.673Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T05:38:16.673Z] ] *** [2026-04-16T05:38:26.063Z] skipping: [an-a01n02] [2026-04-16T05:38:26.063Z] changed: [an-a01n01] [2026-04-16T05:38:26.063Z] [2026-04-16T05:38:26.063Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T05:38:26.063Z] skipping: [an-a01n01] [2026-04-16T05:38:26.063Z] skipping: [an-a01n02] [2026-04-16T05:38:26.063Z] [2026-04-16T05:38:26.063Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T05:38:26.063Z] skipping: [an-a01n01] [2026-04-16T05:38:26.063Z] skipping: [an-a01n02] [2026-04-16T05:38:26.063Z] [2026-04-16T05:38:26.063Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T05:38:26.063Z] skipping: [an-a01n01] [2026-04-16T05:38:26.063Z] skipping: [an-a01n02] [2026-04-16T05:38:26.063Z] [2026-04-16T05:38:26.063Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T05:38:26.063Z] skipping: [an-a01n01] [2026-04-16T05:38:26.063Z] skipping: [an-a01n02] [2026-04-16T05:38:26.063Z] [2026-04-16T05:38:26.063Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T05:38:26.063Z] ] *** [2026-04-16T05:38:27.347Z] skipping: [an-a01n02] [2026-04-16T05:38:27.347Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.999119", "end": "2026-04-16 05:38:26.832627", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 05:38:25.833508", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T05:38:27.347Z] ...ignoring [2026-04-16T05:38:27.347Z] [2026-04-16T05:38:27.347Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T05:38:27.347Z] --ci-test \ [2026-04-16T05:38:27.347Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T05:38:27.347Z] --os almalinux9 \ [2026-04-16T05:38:27.347Z] --cpu 2 --ram 4G \ [2026-04-16T05:38:27.347Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T05:38:27.347Z] --install-media {{ installisouuid }} \ [2026-04-16T05:38:27.347Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T05:38:27.347Z] ] *** [2026-04-16T05:38:36.938Z] skipping: [an-a01n02] [2026-04-16T05:38:36.938Z] changed: [an-a01n01] [2026-04-16T05:38:36.938Z] [2026-04-16T05:38:36.938Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T05:38:36.938Z] skipping: [an-a01n01] [2026-04-16T05:38:36.938Z] skipping: [an-a01n02] [2026-04-16T05:38:36.938Z] [2026-04-16T05:38:36.938Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T05:38:36.938Z] skipping: [an-a01n01] [2026-04-16T05:38:36.938Z] skipping: [an-a01n02] [2026-04-16T05:38:36.938Z] [2026-04-16T05:38:36.938Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T05:38:36.938Z] skipping: [an-a01n01] [2026-04-16T05:38:36.938Z] skipping: [an-a01n02] [2026-04-16T05:38:36.938Z] [2026-04-16T05:38:36.938Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T05:38:36.938Z] skipping: [an-a01n01] [2026-04-16T05:38:36.938Z] skipping: [an-a01n02] [2026-04-16T05:38:36.938Z] [2026-04-16T05:38:36.938Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T05:38:36.938Z] ] *** [2026-04-16T05:38:37.312Z] skipping: [an-a01n02] [2026-04-16T05:38:37.312Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.912528", "end": "2026-04-16 05:38:37.168009", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 05:38:36.255481", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T05:38:37.312Z] ...ignoring [2026-04-16T05:38:37.312Z] [2026-04-16T05:38:37.312Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T05:38:37.312Z] --ci-test \ [2026-04-16T05:38:37.312Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T05:38:37.312Z] --os almalinux9 \ [2026-04-16T05:38:37.312Z] --cpu 2 --ram 4G \ [2026-04-16T05:38:37.312Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T05:38:37.312Z] --install-media {{ installisouuid }} \ [2026-04-16T05:38:37.312Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T05:38:37.312Z] ] *** [2026-04-16T05:38:46.699Z] skipping: [an-a01n02] [2026-04-16T05:38:46.699Z] changed: [an-a01n01] [2026-04-16T05:38:46.699Z] [2026-04-16T05:38:46.699Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T05:38:46.699Z] skipping: [an-a01n01] [2026-04-16T05:38:46.699Z] skipping: [an-a01n02] [2026-04-16T05:38:46.699Z] [2026-04-16T05:38:46.699Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T05:38:46.699Z] skipping: [an-a01n01] [2026-04-16T05:38:46.699Z] skipping: [an-a01n02] [2026-04-16T05:38:46.699Z] [2026-04-16T05:38:46.699Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T05:38:46.699Z] skipping: [an-a01n01] [2026-04-16T05:38:46.699Z] skipping: [an-a01n02] [2026-04-16T05:38:46.699Z] [2026-04-16T05:38:46.699Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T05:38:46.699Z] skipping: [an-a01n01] [2026-04-16T05:38:46.699Z] skipping: [an-a01n02] [2026-04-16T05:38:46.699Z] [2026-04-16T05:38:46.699Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T05:38:46.699Z] ] *** [2026-04-16T05:38:48.542Z] skipping: [an-a01n02] [2026-04-16T05:38:48.542Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.961112", "end": "2026-04-16 05:38:48.066206", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 05:38:47.105094", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T05:38:48.542Z] ...ignoring [2026-04-16T05:38:48.542Z] [2026-04-16T05:38:48.542Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T05:38:48.542Z] --ci-test \ [2026-04-16T05:38:48.542Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T05:38:48.542Z] --os almalinux9 \ [2026-04-16T05:38:48.542Z] --cpu 2 --ram 4G \ [2026-04-16T05:38:48.542Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T05:38:48.542Z] --install-media {{ installisouuid }} \ [2026-04-16T05:38:48.542Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T05:38:48.542Z] ] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] changed: [an-a01n01] [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n01] [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n01] [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n01] [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n01] [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T05:38:59.910Z] ] *** [2026-04-16T05:38:59.910Z] skipping: [an-a01n02] [2026-04-16T05:38:59.910Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.882458", "end": "2026-04-16 05:38:59.650485", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 05:38:58.768027", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T05:38:59.910Z] ...ignoring [2026-04-16T05:38:59.910Z] [2026-04-16T05:38:59.910Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T05:38:59.910Z] --ci-test \ [2026-04-16T05:38:59.910Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T05:38:59.910Z] --os almalinux9 \ [2026-04-16T05:38:59.910Z] --cpu 2 --ram 4G \ [2026-04-16T05:38:59.910Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T05:38:59.910Z] --install-media {{ installisouuid }} \ [2026-04-16T05:38:59.910Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T05:38:59.910Z] ] *** [2026-04-16T05:39:11.426Z] skipping: [an-a01n02] [2026-04-16T05:39:11.426Z] changed: [an-a01n01] [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T05:39:11.426Z] skipping: [an-a01n01] [2026-04-16T05:39:11.426Z] skipping: [an-a01n02] [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T05:39:11.426Z] skipping: [an-a01n01] [2026-04-16T05:39:11.426Z] skipping: [an-a01n02] [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T05:39:11.426Z] skipping: [an-a01n01] [2026-04-16T05:39:11.426Z] skipping: [an-a01n02] [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T05:39:11.426Z] skipping: [an-a01n01] [2026-04-16T05:39:11.426Z] skipping: [an-a01n02] [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-04-16T05:39:11.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T05:39:11.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T05:39:11.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T05:39:11.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T05:39:11.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T05:39:11.426Z] [2026-04-16T05:39:11.426Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T05:39:11.426Z] ] *** [2026-04-16T05:39:39.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-16T05:39:39.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-16T05:39:39.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-16T05:39:39.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-16T05:39:39.939Z] ok: [an-a01n02] [2026-04-16T05:39:39.939Z] ok: [an-a01n01] [2026-04-16T05:39:39.939Z] [2026-04-16T05:39:39.939Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T05:39:39.939Z] ] *** [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-16T05:41:49.587Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-16T05:41:49.587Z] ok: [an-a01n02] [2026-04-16T05:41:49.587Z] ok: [an-a01n01] [2026-04-16T05:41:49.587Z] [2026-04-16T05:41:49.587Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T05:41:49.587Z] ] *** [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-16T05:44:44.483Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left). [2026-04-16T05:44:44.483Z] ok: [an-a01n01] [2026-04-16T05:44:44.483Z] ok: [an-a01n02] [2026-04-16T05:44:44.483Z] [2026-04-16T05:44:44.483Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T05:44:44.483Z] ] *** [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (346 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (346 retries left). [2026-04-16T05:47:54.654Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (345 retries left). [2026-04-16T05:47:54.654Z] ok: [an-a01n01] [2026-04-16T05:47:54.654Z] ok: [an-a01n02] [2026-04-16T05:47:54.654Z] [2026-04-16T05:47:54.654Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T05:47:54.654Z] ] *** [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-16T05:51:04.638Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-04-16T05:51:04.638Z] ok: [an-a01n01] [2026-04-16T05:51:04.638Z] ok: [an-a01n02] [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-04-16T05:51:04.638Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T05:51:04.638Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T05:51:04.638Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T05:51:04.638Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T05:51:04.638Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T05:51:04.638Z] ] *** [2026-04-16T05:51:04.638Z] ok: [an-a01n02] [2026-04-16T05:51:04.638Z] ok: [an-a01n01] [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T05:51:04.638Z] ] *** [2026-04-16T05:51:04.638Z] ok: [an-a01n02] [2026-04-16T05:51:04.638Z] ok: [an-a01n01] [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T05:51:04.638Z] ] *** [2026-04-16T05:51:04.638Z] ok: [an-a01n02] [2026-04-16T05:51:04.638Z] ok: [an-a01n01] [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T05:51:04.638Z] ] *** [2026-04-16T05:51:04.638Z] ok: [an-a01n02] [2026-04-16T05:51:04.638Z] ok: [an-a01n01] [2026-04-16T05:51:04.638Z] [2026-04-16T05:51:04.638Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T05:51:04.638Z] ] *** [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-04-16T05:51:45.562Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-04-16T05:51:45.562Z] ok: [an-a01n02] [2026-04-16T05:51:45.562Z] ok: [an-a01n01] [2026-04-16T05:51:45.562Z] [2026-04-16T05:51:45.562Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-04-16T05:51:45.562Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T05:51:45.562Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T05:51:45.562Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T05:51:45.562Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T05:51:45.562Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T05:51:45.562Z] [2026-04-16T05:51:45.562Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T05:51:45.562Z] ] *** [2026-04-16T05:51:45.562Z] ok: [an-a01n02] [2026-04-16T05:51:45.562Z] ok: [an-a01n01] [2026-04-16T05:51:45.562Z] [2026-04-16T05:51:45.562Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T05:51:45.562Z] ] *** [2026-04-16T05:51:45.562Z] ok: [an-a01n02] [2026-04-16T05:51:45.562Z] ok: [an-a01n01] [2026-04-16T05:51:45.562Z] [2026-04-16T05:51:45.562Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T05:51:45.562Z] ] *** [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-16T05:52:44.386Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-16T05:52:44.386Z] ok: [an-a01n02] [2026-04-16T05:52:44.386Z] ok: [an-a01n01] [2026-04-16T05:52:44.386Z] [2026-04-16T05:52:44.386Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T05:52:44.386Z] ] *** [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (346 retries left). [2026-04-16T05:56:09.620Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (346 retries left). [2026-04-16T05:56:09.620Z] ok: [an-a01n02] [2026-04-16T05:56:09.620Z] ok: [an-a01n01] [2026-04-16T05:56:09.620Z] [2026-04-16T05:56:09.620Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T05:56:09.620Z] ] *** [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-04-16T05:59:49.904Z] ok: [an-a01n01] [2026-04-16T05:59:49.904Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left). [2026-04-16T05:59:49.904Z] ok: [an-a01n02] [2026-04-16T05:59:49.904Z] [2026-04-16T05:59:49.904Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-04-16T05:59:49.904Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n02, an-a01n01 => (item=1) [2026-04-16T05:59:49.904Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n02, an-a01n01 => (item=2) [2026-04-16T05:59:49.904Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n02, an-a01n01 => (item=3) [2026-04-16T05:59:49.904Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n02, an-a01n01 => (item=4) [2026-04-16T05:59:49.904Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n02, an-a01n01 => (item=5) [2026-04-16T05:59:49.904Z] [2026-04-16T05:59:49.904Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T05:59:49.904Z] ] *** [2026-04-16T05:59:50.827Z] ok: [an-a01n01] [2026-04-16T05:59:50.827Z] ok: [an-a01n02] [2026-04-16T05:59:50.827Z] [2026-04-16T05:59:50.827Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T05:59:50.827Z] ] *** [2026-04-16T05:59:58.550Z] ok: [an-a01n02] [2026-04-16T05:59:58.550Z] ok: [an-a01n01] [2026-04-16T05:59:58.550Z] [2026-04-16T05:59:58.550Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T05:59:58.550Z] ] *** [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-16T06:03:08.573Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-16T06:03:08.573Z] ok: [an-a01n02] [2026-04-16T06:03:08.573Z] ok: [an-a01n01] [2026-04-16T06:03:08.573Z] [2026-04-16T06:03:08.573Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:03:08.573Z] ] *** [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left). [2026-04-16T06:05:48.431Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left). [2026-04-16T06:05:48.431Z] ok: [an-a01n02] [2026-04-16T06:05:48.431Z] ok: [an-a01n01] [2026-04-16T06:05:48.431Z] [2026-04-16T06:05:48.431Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:05:48.431Z] ] *** [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-04-16T06:08:28.187Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-04-16T06:08:28.187Z] ok: [an-a01n02] [2026-04-16T06:08:28.187Z] ok: [an-a01n01] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-04-16T06:08:28.187Z] ok: [an-a01n01] [2026-04-16T06:08:28.187Z] ok: [an-a01n02] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:08:28.187Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:08:28.187Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:08:28.187Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:08:28.187Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:08:28.187Z] echo "pcs:" [2026-04-16T06:08:28.187Z] echo "$pcsstatus" [2026-04-16T06:08:28.187Z] exit 1 [2026-04-16T06:08:28.187Z] fi [2026-04-16T06:08:28.187Z] exit 0 [2026-04-16T06:08:28.187Z] ] *** [2026-04-16T06:08:28.187Z] ok: [an-a01n02] [2026-04-16T06:08:28.187Z] ok: [an-a01n01] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:08:28.187Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:08:28.187Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:08:28.187Z] ret=0 [2026-04-16T06:08:28.187Z] if [ "$servers" != "1" ]; then [2026-04-16T06:08:28.187Z] echo "Not all servers are running on the same node!" [2026-04-16T06:08:28.187Z] echo "$pcsstatus" [2026-04-16T06:08:28.187Z] ret=1 [2026-04-16T06:08:28.187Z] fi [2026-04-16T06:08:28.187Z] echo "$hypervisor" [2026-04-16T06:08:28.187Z] exit $ret [2026-04-16T06:08:28.187Z] ] *** [2026-04-16T06:08:28.187Z] ok: [an-a01n02] [2026-04-16T06:08:28.187Z] ok: [an-a01n01] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T06:08:28.187Z] ok: [an-a01n01] [2026-04-16T06:08:28.187Z] ok: [an-a01n02] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T06:08:28.187Z] ] ********** [2026-04-16T06:08:28.187Z] skipping: [an-a01n02] [2026-04-16T06:08:28.187Z] changed: [an-a01n01] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T06:08:28.187Z] skipping: [an-a01n01] [2026-04-16T06:08:28.187Z] skipping: [an-a01n02] [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T06:08:28.187Z] skipping: [an-a01n01] [2026-04-16T06:08:28.187Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T06:08:28.187Z] [2026-04-16T06:08:28.187Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T06:08:28.187Z] ] *** [2026-04-16T06:09:38.541Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T06:09:38.541Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T06:09:38.541Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T06:09:38.541Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T06:09:38.541Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T06:09:38.541Z] ok: [an-a01n02] [2026-04-16T06:09:38.541Z] [2026-04-16T06:09:38.541Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:09:38.541Z] skipping: [an-a01n01] [2026-04-16T06:09:38.541Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T06:09:38.541Z] [2026-04-16T06:09:38.541Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:09:38.541Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:09:38.541Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:09:38.541Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:09:38.541Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:09:38.541Z] echo "pcs:" [2026-04-16T06:09:38.541Z] echo "$pcsstatus" [2026-04-16T06:09:38.541Z] exit 1 [2026-04-16T06:09:38.541Z] fi [2026-04-16T06:09:38.541Z] exit 0 [2026-04-16T06:09:38.541Z] ] *** [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-04-16T06:11:17.771Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-04-16T06:11:17.771Z] ok: [an-a01n02] [2026-04-16T06:11:17.771Z] [2026-04-16T06:11:17.771Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:11:17.771Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:11:17.771Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:11:17.771Z] ret=0 [2026-04-16T06:11:17.771Z] if [ "$servers" != "1" ]; then [2026-04-16T06:11:17.771Z] echo "Not all servers are running on the same node!" [2026-04-16T06:11:17.771Z] echo "$pcsstatus" [2026-04-16T06:11:17.771Z] ret=1 [2026-04-16T06:11:17.771Z] fi [2026-04-16T06:11:17.771Z] echo "$hypervisor" [2026-04-16T06:11:17.771Z] exit $ret [2026-04-16T06:11:17.771Z] ] *** [2026-04-16T06:11:17.771Z] ok: [an-a01n02] [2026-04-16T06:11:17.771Z] [2026-04-16T06:11:17.771Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] => (item=1) [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] => (item=2) [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] => (item=3) [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] => (item=4) [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] => (item=5) [2026-04-16T06:11:17.771Z] skipping: [an-a01n01] [2026-04-16T06:11:17.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T06:11:17.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T06:11:17.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T06:11:17.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T06:11:17.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T06:11:17.771Z] [2026-04-16T06:11:17.771Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:11:17.771Z] ] *** [2026-04-16T06:11:19.610Z] ok: [an-a01n02] [2026-04-16T06:11:19.610Z] [2026-04-16T06:11:19.610Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:11:19.610Z] ] *** [2026-04-16T06:11:20.410Z] ok: [an-a01n02] [2026-04-16T06:11:20.410Z] [2026-04-16T06:11:20.410Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:11:20.410Z] ] *** [2026-04-16T06:11:21.210Z] ok: [an-a01n02] [2026-04-16T06:11:21.210Z] [2026-04-16T06:11:21.210Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:11:21.210Z] ] *** [2026-04-16T06:11:22.493Z] ok: [an-a01n02] [2026-04-16T06:11:22.493Z] [2026-04-16T06:11:22.493Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:11:22.493Z] ] *** [2026-04-16T06:11:23.293Z] ok: [an-a01n02] [2026-04-16T06:11:23.293Z] [2026-04-16T06:11:23.293Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T06:11:23.293Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:11:23.293Z] [2026-04-16T06:11:23.293Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T06:14:48.554Z] ok: [an-a01n01 -> localhost] [2026-04-16T06:14:48.554Z] ok: [an-a01n02 -> localhost] [2026-04-16T06:14:48.554Z] [2026-04-16T06:14:48.554Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:14:48.554Z] [2026-04-16T06:14:48.554Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:14:48.554Z] ] *** [2026-04-16T06:14:48.554Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:14:48.554Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:14:48.554Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:14:48.554Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:14:48.554Z] [2026-04-16T06:14:48.554Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:14:48.554Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:14:48.554Z] [2026-04-16T06:14:48.554Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:14:48.554Z] ] *** [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-04-16T06:15:37.549Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:15:37.549Z] ] *** [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:15:37.549Z] ] *** [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:15:37.549Z] ] *** [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:15:37.549Z] ] *** [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-04-16T06:15:37.549Z] ok: [an-a01n01] [2026-04-16T06:15:37.549Z] ok: [an-a01n02] [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:15:37.549Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:15:37.549Z] [2026-04-16T06:15:37.549Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:15:37.549Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:15:37.549Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:15:37.549Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:15:37.549Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:15:37.549Z] echo "pcs:" [2026-04-16T06:15:37.549Z] echo "$pcsstatus" [2026-04-16T06:15:37.549Z] exit 1 [2026-04-16T06:15:37.549Z] fi [2026-04-16T06:15:37.549Z] exit 0 [2026-04-16T06:15:37.549Z] ] *** [2026-04-16T06:15:38.470Z] ok: [an-a01n02] [2026-04-16T06:15:38.470Z] ok: [an-a01n01] [2026-04-16T06:15:38.470Z] [2026-04-16T06:15:38.470Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:15:38.470Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:15:38.470Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:15:38.470Z] ret=0 [2026-04-16T06:15:38.470Z] if [ "$servers" != "1" ]; then [2026-04-16T06:15:38.470Z] echo "Not all servers are running on the same node!" [2026-04-16T06:15:38.470Z] echo "$pcsstatus" [2026-04-16T06:15:38.470Z] ret=1 [2026-04-16T06:15:38.470Z] fi [2026-04-16T06:15:38.470Z] echo "$hypervisor" [2026-04-16T06:15:38.470Z] exit $ret [2026-04-16T06:15:38.470Z] ] *** [2026-04-16T06:15:40.311Z] ok: [an-a01n01] [2026-04-16T06:15:40.311Z] ok: [an-a01n02] [2026-04-16T06:15:40.311Z] [2026-04-16T06:15:40.311Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T06:15:40.311Z] ok: [an-a01n01] [2026-04-16T06:15:40.311Z] ok: [an-a01n02] [2026-04-16T06:15:40.311Z] [2026-04-16T06:15:40.311Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T06:15:40.684Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:15:40.684Z] [2026-04-16T06:15:40.684Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T06:15:40.684Z] ] ********** [2026-04-16T06:15:41.483Z] skipping: [an-a01n01] [2026-04-16T06:15:41.483Z] changed: [an-a01n02] [2026-04-16T06:15:41.483Z] [2026-04-16T06:15:41.483Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T06:15:41.483Z] skipping: [an-a01n01] [2026-04-16T06:15:41.483Z] skipping: [an-a01n02] [2026-04-16T06:15:41.483Z] [2026-04-16T06:15:41.483Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T06:15:41.483Z] skipping: [an-a01n02] [2026-04-16T06:15:41.483Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-16T06:15:41.483Z] [2026-04-16T06:15:41.483Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T06:15:41.483Z] ] *** [2026-04-16T06:16:51.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-16T06:16:51.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-16T06:16:51.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-16T06:16:51.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-16T06:16:51.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-16T06:16:51.717Z] ok: [an-a01n01] [2026-04-16T06:16:51.717Z] [2026-04-16T06:16:51.717Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:16:51.717Z] skipping: [an-a01n02] [2026-04-16T06:16:51.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-16T06:16:51.717Z] [2026-04-16T06:16:51.717Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:16:51.717Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:16:51.717Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:16:51.717Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:16:51.717Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:16:51.717Z] echo "pcs:" [2026-04-16T06:16:51.717Z] echo "$pcsstatus" [2026-04-16T06:16:51.717Z] exit 1 [2026-04-16T06:16:51.717Z] fi [2026-04-16T06:16:51.717Z] exit 0 [2026-04-16T06:16:51.717Z] ] *** [2026-04-16T06:17:50.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T06:17:50.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T06:17:50.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T06:17:50.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T06:17:50.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T06:17:50.509Z] ok: [an-a01n01] [2026-04-16T06:17:50.509Z] [2026-04-16T06:17:50.509Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:17:50.509Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:17:50.509Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:17:50.509Z] ret=0 [2026-04-16T06:17:50.509Z] if [ "$servers" != "1" ]; then [2026-04-16T06:17:50.509Z] echo "Not all servers are running on the same node!" [2026-04-16T06:17:50.509Z] echo "$pcsstatus" [2026-04-16T06:17:50.509Z] ret=1 [2026-04-16T06:17:50.509Z] fi [2026-04-16T06:17:50.509Z] echo "$hypervisor" [2026-04-16T06:17:50.509Z] exit $ret [2026-04-16T06:17:50.509Z] ] *** [2026-04-16T06:17:50.509Z] ok: [an-a01n01] [2026-04-16T06:17:50.509Z] [2026-04-16T06:17:50.509Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] => (item=1) [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] => (item=2) [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] => (item=3) [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] => (item=4) [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] => (item=5) [2026-04-16T06:17:50.509Z] skipping: [an-a01n02] [2026-04-16T06:17:50.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-16T06:17:50.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-16T06:17:50.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-16T06:17:50.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-16T06:17:50.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-16T06:17:50.509Z] [2026-04-16T06:17:50.509Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:17:50.509Z] ] *** [2026-04-16T06:18:07.048Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T06:18:07.048Z] ok: [an-a01n01] [2026-04-16T06:18:07.048Z] [2026-04-16T06:18:07.048Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:18:07.048Z] ] *** [2026-04-16T06:18:20.790Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-04-16T06:18:20.790Z] ok: [an-a01n01] [2026-04-16T06:18:20.790Z] [2026-04-16T06:18:20.790Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:18:20.790Z] ] *** [2026-04-16T06:18:20.790Z] ok: [an-a01n01] [2026-04-16T06:18:20.790Z] [2026-04-16T06:18:20.790Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:18:20.790Z] ] *** [2026-04-16T06:18:21.164Z] ok: [an-a01n01] [2026-04-16T06:18:21.164Z] [2026-04-16T06:18:21.164Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:18:21.164Z] ] *** [2026-04-16T06:18:21.962Z] ok: [an-a01n01] [2026-04-16T06:18:21.962Z] [2026-04-16T06:18:21.962Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T06:18:22.336Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:18:22.336Z] [2026-04-16T06:18:22.336Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T06:21:47.559Z] ok: [an-a01n01 -> localhost] [2026-04-16T06:21:47.559Z] ok: [an-a01n02 -> localhost] [2026-04-16T06:21:47.559Z] [2026-04-16T06:21:47.559Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:21:47.559Z] [2026-04-16T06:21:47.559Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:21:47.559Z] ] *** [2026-04-16T06:21:47.559Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:21:47.559Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:21:47.559Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:21:47.559Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:21:47.559Z] [2026-04-16T06:21:47.559Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:21:47.559Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:21:47.559Z] [2026-04-16T06:21:47.559Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:21:47.559Z] ] *** [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T06:22:11.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T06:22:11.345Z] ok: [an-a01n02] [2026-04-16T06:22:11.345Z] ok: [an-a01n01] [2026-04-16T06:22:11.345Z] [2026-04-16T06:22:11.345Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:22:11.345Z] ] *** [2026-04-16T06:22:35.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-04-16T06:22:35.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-04-16T06:22:35.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-04-16T06:22:35.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-04-16T06:22:35.251Z] ok: [an-a01n02] [2026-04-16T06:22:35.251Z] ok: [an-a01n01] [2026-04-16T06:22:35.251Z] [2026-04-16T06:22:35.251Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:22:35.251Z] ] *** [2026-04-16T06:22:35.251Z] ok: [an-a01n02] [2026-04-16T06:22:35.251Z] ok: [an-a01n01] [2026-04-16T06:22:35.251Z] [2026-04-16T06:22:35.251Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:22:35.251Z] ] *** [2026-04-16T06:22:35.251Z] ok: [an-a01n01] [2026-04-16T06:22:35.251Z] ok: [an-a01n02] [2026-04-16T06:22:35.251Z] [2026-04-16T06:22:35.251Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:22:35.251Z] ] *** [2026-04-16T06:22:35.251Z] ok: [an-a01n01] [2026-04-16T06:22:35.251Z] ok: [an-a01n02] [2026-04-16T06:22:35.251Z] [2026-04-16T06:22:35.251Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-04-16T06:22:35.251Z] ok: [an-a01n01] [2026-04-16T06:22:35.251Z] ok: [an-a01n02] [2026-04-16T06:22:35.251Z] [2026-04-16T06:22:35.251Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:22:35.625Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:22:35.625Z] [2026-04-16T06:22:35.625Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:22:35.625Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:22:35.625Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:22:35.625Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:22:35.625Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:22:35.625Z] echo "pcs:" [2026-04-16T06:22:35.625Z] echo "$pcsstatus" [2026-04-16T06:22:35.625Z] exit 1 [2026-04-16T06:22:35.625Z] fi [2026-04-16T06:22:35.625Z] exit 0 [2026-04-16T06:22:35.625Z] ] *** [2026-04-16T06:22:38.112Z] ok: [an-a01n02] [2026-04-16T06:22:38.112Z] ok: [an-a01n01] [2026-04-16T06:22:38.112Z] [2026-04-16T06:22:38.112Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:22:38.112Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:22:38.112Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:22:38.112Z] ret=0 [2026-04-16T06:22:38.112Z] if [ "$servers" != "1" ]; then [2026-04-16T06:22:38.112Z] echo "Not all servers are running on the same node!" [2026-04-16T06:22:38.112Z] echo "$pcsstatus" [2026-04-16T06:22:38.112Z] ret=1 [2026-04-16T06:22:38.112Z] fi [2026-04-16T06:22:38.112Z] echo "$hypervisor" [2026-04-16T06:22:38.112Z] exit $ret [2026-04-16T06:22:38.112Z] ] *** [2026-04-16T06:22:39.950Z] ok: [an-a01n02] [2026-04-16T06:22:39.950Z] ok: [an-a01n01] [2026-04-16T06:22:39.950Z] [2026-04-16T06:22:39.950Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T06:22:39.950Z] ok: [an-a01n01] [2026-04-16T06:22:39.950Z] ok: [an-a01n02] [2026-04-16T06:22:39.950Z] [2026-04-16T06:22:39.950Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T06:22:39.950Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:22:39.950Z] [2026-04-16T06:22:39.950Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T06:22:39.950Z] ] ********** [2026-04-16T06:22:40.748Z] skipping: [an-a01n02] [2026-04-16T06:22:40.748Z] changed: [an-a01n01] [2026-04-16T06:22:40.748Z] [2026-04-16T06:22:40.748Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T06:22:40.748Z] skipping: [an-a01n01] [2026-04-16T06:22:40.748Z] skipping: [an-a01n02] [2026-04-16T06:22:40.748Z] [2026-04-16T06:22:40.748Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T06:22:41.122Z] skipping: [an-a01n01] [2026-04-16T06:22:41.122Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T06:22:41.122Z] [2026-04-16T06:22:41.122Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T06:22:41.122Z] ] *** [2026-04-16T06:23:51.509Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T06:23:51.509Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T06:23:51.509Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T06:23:51.509Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T06:23:51.509Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T06:23:51.509Z] ok: [an-a01n02] [2026-04-16T06:23:51.509Z] [2026-04-16T06:23:51.510Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:23:51.510Z] skipping: [an-a01n01] [2026-04-16T06:23:51.510Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T06:23:51.510Z] [2026-04-16T06:23:51.510Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:23:51.510Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:23:51.510Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:23:51.510Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:23:51.510Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:23:51.510Z] echo "pcs:" [2026-04-16T06:23:51.510Z] echo "$pcsstatus" [2026-04-16T06:23:51.510Z] exit 1 [2026-04-16T06:23:51.510Z] fi [2026-04-16T06:23:51.510Z] exit 0 [2026-04-16T06:23:51.510Z] ] *** [2026-04-16T06:24:50.168Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T06:24:50.168Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T06:24:50.168Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T06:24:50.168Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T06:24:50.168Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T06:24:50.168Z] ok: [an-a01n02] [2026-04-16T06:24:50.168Z] [2026-04-16T06:24:50.168Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:24:50.168Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:24:50.168Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:24:50.168Z] ret=0 [2026-04-16T06:24:50.168Z] if [ "$servers" != "1" ]; then [2026-04-16T06:24:50.168Z] echo "Not all servers are running on the same node!" [2026-04-16T06:24:50.168Z] echo "$pcsstatus" [2026-04-16T06:24:50.168Z] ret=1 [2026-04-16T06:24:50.168Z] fi [2026-04-16T06:24:50.168Z] echo "$hypervisor" [2026-04-16T06:24:50.168Z] exit $ret [2026-04-16T06:24:50.168Z] ] *** [2026-04-16T06:24:50.168Z] ok: [an-a01n02] [2026-04-16T06:24:50.168Z] [2026-04-16T06:24:50.168Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] => (item=1) [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] => (item=2) [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] => (item=3) [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] => (item=4) [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] => (item=5) [2026-04-16T06:24:50.168Z] skipping: [an-a01n01] [2026-04-16T06:24:50.168Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T06:24:50.168Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T06:24:50.168Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T06:24:50.168Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T06:24:50.168Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T06:24:50.168Z] [2026-04-16T06:24:50.168Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:24:50.168Z] ] *** [2026-04-16T06:25:04.001Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T06:25:04.001Z] ok: [an-a01n02] [2026-04-16T06:25:04.001Z] [2026-04-16T06:25:04.001Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:25:04.001Z] ] *** [2026-04-16T06:25:04.801Z] ok: [an-a01n02] [2026-04-16T06:25:04.801Z] [2026-04-16T06:25:04.801Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:25:04.801Z] ] *** [2026-04-16T06:25:05.600Z] ok: [an-a01n02] [2026-04-16T06:25:05.600Z] [2026-04-16T06:25:05.600Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:25:05.600Z] ] *** [2026-04-16T06:25:06.399Z] ok: [an-a01n02] [2026-04-16T06:25:06.399Z] [2026-04-16T06:25:06.399Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:25:06.399Z] ] *** [2026-04-16T06:25:07.197Z] ok: [an-a01n02] [2026-04-16T06:25:07.197Z] [2026-04-16T06:25:07.197Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T06:25:07.571Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:25:07.571Z] [2026-04-16T06:25:07.571Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01 -> localhost] [2026-04-16T06:28:32.691Z] ok: [an-a01n02 -> localhost] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:28:32.691Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:28:32.691Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:28:32.691Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:28:32.691Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:28:32.691Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:28:32.691Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:28:32.691Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:28:32.691Z] echo "pcs:" [2026-04-16T06:28:32.691Z] echo "$pcsstatus" [2026-04-16T06:28:32.691Z] exit 1 [2026-04-16T06:28:32.691Z] fi [2026-04-16T06:28:32.691Z] exit 0 [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:28:32.691Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:28:32.691Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:28:32.691Z] ret=0 [2026-04-16T06:28:32.691Z] if [ "$servers" != "1" ]; then [2026-04-16T06:28:32.691Z] echo "Not all servers are running on the same node!" [2026-04-16T06:28:32.691Z] echo "$pcsstatus" [2026-04-16T06:28:32.691Z] ret=1 [2026-04-16T06:28:32.691Z] fi [2026-04-16T06:28:32.691Z] echo "$hypervisor" [2026-04-16T06:28:32.691Z] exit $ret [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T06:28:32.691Z] ok: [an-a01n01] [2026-04-16T06:28:32.691Z] ok: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T06:28:32.691Z] ] ********** [2026-04-16T06:28:32.691Z] skipping: [an-a01n01] [2026-04-16T06:28:32.691Z] changed: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T06:28:32.691Z] skipping: [an-a01n01] [2026-04-16T06:28:32.691Z] skipping: [an-a01n02] [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T06:28:32.691Z] skipping: [an-a01n02] [2026-04-16T06:28:32.691Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-16T06:28:32.691Z] [2026-04-16T06:28:32.691Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T06:28:32.691Z] ] *** [2026-04-16T06:29:42.926Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-16T06:29:42.926Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-16T06:29:42.926Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-16T06:29:42.926Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-16T06:29:42.926Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-16T06:29:42.926Z] ok: [an-a01n01] [2026-04-16T06:29:42.926Z] [2026-04-16T06:29:42.926Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:29:42.926Z] skipping: [an-a01n02] [2026-04-16T06:29:42.926Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-16T06:29:42.926Z] [2026-04-16T06:29:42.926Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:29:42.926Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:29:42.926Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:29:42.926Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:29:42.926Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:29:42.926Z] echo "pcs:" [2026-04-16T06:29:42.926Z] echo "$pcsstatus" [2026-04-16T06:29:42.926Z] exit 1 [2026-04-16T06:29:42.926Z] fi [2026-04-16T06:29:42.926Z] exit 0 [2026-04-16T06:29:42.926Z] ] *** [2026-04-16T06:30:41.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T06:30:41.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T06:30:41.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T06:30:41.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T06:30:41.717Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T06:30:41.717Z] ok: [an-a01n01] [2026-04-16T06:30:41.717Z] [2026-04-16T06:30:41.717Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:30:41.717Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:30:41.717Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:30:41.717Z] ret=0 [2026-04-16T06:30:41.717Z] if [ "$servers" != "1" ]; then [2026-04-16T06:30:41.717Z] echo "Not all servers are running on the same node!" [2026-04-16T06:30:41.717Z] echo "$pcsstatus" [2026-04-16T06:30:41.717Z] ret=1 [2026-04-16T06:30:41.717Z] fi [2026-04-16T06:30:41.717Z] echo "$hypervisor" [2026-04-16T06:30:41.717Z] exit $ret [2026-04-16T06:30:41.717Z] ] *** [2026-04-16T06:30:41.717Z] ok: [an-a01n01] [2026-04-16T06:30:41.717Z] [2026-04-16T06:30:41.717Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] => (item=1) [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] => (item=2) [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] => (item=3) [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] => (item=4) [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] => (item=5) [2026-04-16T06:30:41.717Z] skipping: [an-a01n02] [2026-04-16T06:30:41.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-16T06:30:41.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-16T06:30:41.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-16T06:30:41.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-16T06:30:41.717Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-16T06:30:41.717Z] [2026-04-16T06:30:41.717Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:30:41.717Z] ] *** [2026-04-16T06:30:53.088Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T06:30:53.088Z] ok: [an-a01n01] [2026-04-16T06:30:53.088Z] [2026-04-16T06:30:53.088Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:30:53.088Z] ] *** [2026-04-16T06:30:53.887Z] ok: [an-a01n01] [2026-04-16T06:30:53.887Z] [2026-04-16T06:30:53.887Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:30:53.887Z] ] *** [2026-04-16T06:30:54.686Z] ok: [an-a01n01] [2026-04-16T06:30:54.686Z] [2026-04-16T06:30:54.686Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:30:54.686Z] ] *** [2026-04-16T06:30:55.485Z] ok: [an-a01n01] [2026-04-16T06:30:55.485Z] [2026-04-16T06:30:55.485Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:30:55.485Z] ] *** [2026-04-16T06:30:55.863Z] ok: [an-a01n01] [2026-04-16T06:30:55.863Z] [2026-04-16T06:30:55.863Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T06:30:56.356Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:30:56.356Z] [2026-04-16T06:30:56.356Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n01 -> localhost] [2026-04-16T06:34:21.478Z] ok: [an-a01n02 -> localhost] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:34:21.478Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:34:21.478Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:34:21.478Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:34:21.478Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:34:21.478Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:34:21.478Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:34:21.478Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:34:21.478Z] echo "pcs:" [2026-04-16T06:34:21.478Z] echo "$pcsstatus" [2026-04-16T06:34:21.478Z] exit 1 [2026-04-16T06:34:21.478Z] fi [2026-04-16T06:34:21.478Z] exit 0 [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:34:21.478Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:34:21.478Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:34:21.478Z] ret=0 [2026-04-16T06:34:21.478Z] if [ "$servers" != "1" ]; then [2026-04-16T06:34:21.478Z] echo "Not all servers are running on the same node!" [2026-04-16T06:34:21.478Z] echo "$pcsstatus" [2026-04-16T06:34:21.478Z] ret=1 [2026-04-16T06:34:21.478Z] fi [2026-04-16T06:34:21.478Z] echo "$hypervisor" [2026-04-16T06:34:21.478Z] exit $ret [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T06:34:21.478Z] ok: [an-a01n01] [2026-04-16T06:34:21.478Z] ok: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T06:34:21.478Z] ] ********** [2026-04-16T06:34:21.478Z] skipping: [an-a01n02] [2026-04-16T06:34:21.478Z] changed: [an-a01n01] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T06:34:21.478Z] skipping: [an-a01n01] [2026-04-16T06:34:21.478Z] skipping: [an-a01n02] [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T06:34:21.478Z] skipping: [an-a01n01] [2026-04-16T06:34:21.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T06:34:21.478Z] [2026-04-16T06:34:21.478Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T06:34:21.478Z] ] *** [2026-04-16T06:35:31.845Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T06:35:31.845Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T06:35:31.845Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T06:35:31.845Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T06:35:31.845Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T06:35:31.845Z] ok: [an-a01n02] [2026-04-16T06:35:31.845Z] [2026-04-16T06:35:31.845Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T06:35:31.845Z] skipping: [an-a01n01] [2026-04-16T06:35:31.845Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T06:35:31.845Z] [2026-04-16T06:35:31.845Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:35:31.845Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:35:31.845Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:35:31.845Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:35:31.845Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:35:31.845Z] echo "pcs:" [2026-04-16T06:35:31.845Z] echo "$pcsstatus" [2026-04-16T06:35:31.845Z] exit 1 [2026-04-16T06:35:31.845Z] fi [2026-04-16T06:35:31.845Z] exit 0 [2026-04-16T06:35:31.845Z] ] *** [2026-04-16T06:36:30.624Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T06:36:30.624Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T06:36:30.624Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T06:36:30.624Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T06:36:30.624Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T06:36:30.624Z] ok: [an-a01n02] [2026-04-16T06:36:30.624Z] [2026-04-16T06:36:30.624Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:36:30.624Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:36:30.624Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:36:30.624Z] ret=0 [2026-04-16T06:36:30.624Z] if [ "$servers" != "1" ]; then [2026-04-16T06:36:30.624Z] echo "Not all servers are running on the same node!" [2026-04-16T06:36:30.624Z] echo "$pcsstatus" [2026-04-16T06:36:30.624Z] ret=1 [2026-04-16T06:36:30.624Z] fi [2026-04-16T06:36:30.624Z] echo "$hypervisor" [2026-04-16T06:36:30.624Z] exit $ret [2026-04-16T06:36:30.624Z] ] *** [2026-04-16T06:36:30.624Z] ok: [an-a01n02] [2026-04-16T06:36:30.624Z] [2026-04-16T06:36:30.624Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] => (item=1) [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] => (item=2) [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] => (item=3) [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] => (item=4) [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] => (item=5) [2026-04-16T06:36:30.624Z] skipping: [an-a01n01] [2026-04-16T06:36:30.624Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T06:36:30.624Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T06:36:30.624Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T06:36:30.624Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T06:36:30.624Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T06:36:30.624Z] [2026-04-16T06:36:30.624Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:36:30.624Z] ] *** [2026-04-16T06:36:41.994Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T06:36:41.994Z] ok: [an-a01n02] [2026-04-16T06:36:41.994Z] [2026-04-16T06:36:41.994Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:36:41.994Z] ] *** [2026-04-16T06:36:42.794Z] ok: [an-a01n02] [2026-04-16T06:36:42.794Z] [2026-04-16T06:36:42.794Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:36:42.794Z] ] *** [2026-04-16T06:36:43.169Z] ok: [an-a01n02] [2026-04-16T06:36:43.169Z] [2026-04-16T06:36:43.169Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:36:43.169Z] ] *** [2026-04-16T06:36:43.967Z] ok: [an-a01n02] [2026-04-16T06:36:43.967Z] [2026-04-16T06:36:43.967Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T06:36:43.967Z] ] *** [2026-04-16T06:36:44.764Z] ok: [an-a01n02] [2026-04-16T06:36:44.764Z] [2026-04-16T06:36:44.764Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T06:36:44.764Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T06:36:44.764Z] [2026-04-16T06:36:44.764Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n01 -> localhost] [2026-04-16T06:40:10.002Z] ok: [an-a01n02 -> localhost] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:40:10.002Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:40:10.002Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:40:10.002Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] PLAY [basic server tests] ****************************************************** [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T06:40:10.002Z] ok: [an-a01n01] [2026-04-16T06:40:10.002Z] ok: [an-a01n02] [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:10.002Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:40:10.002Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:40:10.002Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:40:10.002Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T06:40:10.002Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:10.002Z] [2026-04-16T06:40:10.002Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T06:40:10.002Z] if [ -z "$storage" ]; then [2026-04-16T06:40:10.002Z] echo "Unable to detect shared storage" [2026-04-16T06:40:10.002Z] exit 1 [2026-04-16T06:40:10.002Z] fi [2026-04-16T06:40:10.002Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T06:40:10.002Z] ] *** [2026-04-16T06:40:16.320Z] changed: [an-a01n01] [2026-04-16T06:40:16.320Z] changed: [an-a01n02] [2026-04-16T06:40:16.320Z] [2026-04-16T06:40:16.320Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T06:40:16.320Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:16.320Z] [2026-04-16T06:40:16.320Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T06:40:16.320Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T06:40:16.320Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T06:40:16.320Z] echo "Unable to detect all deploy iso" [2026-04-16T06:40:16.320Z] echo "Detected isos" [2026-04-16T06:40:16.320Z] echo "$isos" [2026-04-16T06:40:16.320Z] exit 1 [2026-04-16T06:40:16.320Z] fi [2026-04-16T06:40:16.320Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T06:40:16.320Z] if [ -z "$installiso" ]; then [2026-04-16T06:40:16.320Z] echo "Unable to detect install iso" [2026-04-16T06:40:16.320Z] echo "Detected isos" [2026-04-16T06:40:16.320Z] echo "$isos" [2026-04-16T06:40:16.320Z] exit 1 [2026-04-16T06:40:16.320Z] fi [2026-04-16T06:40:16.320Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T06:40:16.320Z] ] *** [2026-04-16T06:40:21.563Z] changed: [an-a01n01] [2026-04-16T06:40:21.563Z] changed: [an-a01n02] [2026-04-16T06:40:21.563Z] [2026-04-16T06:40:21.563Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T06:40:21.563Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:21.563Z] [2026-04-16T06:40:21.563Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:40:21.563Z] ] *** [2026-04-16T06:40:22.363Z] skipping: [an-a01n02] [2026-04-16T06:40:22.363Z] ok: [an-a01n01] [2026-04-16T06:40:22.363Z] [2026-04-16T06:40:22.363Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T06:40:22.363Z] --ci-test \ [2026-04-16T06:40:22.363Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T06:40:22.363Z] --os almalinux9 \ [2026-04-16T06:40:22.363Z] --cpu 2 --ram 4G \ [2026-04-16T06:40:22.363Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T06:40:22.363Z] --install-media {{ installisouuid }} \ [2026-04-16T06:40:22.363Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T06:40:22.363Z] ] *** [2026-04-16T06:40:22.363Z] skipping: [an-a01n01] [2026-04-16T06:40:22.363Z] skipping: [an-a01n02] [2026-04-16T06:40:22.363Z] [2026-04-16T06:40:22.363Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T06:40:22.737Z] skipping: [an-a01n01] [2026-04-16T06:40:22.737Z] skipping: [an-a01n02] [2026-04-16T06:40:22.737Z] [2026-04-16T06:40:22.737Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:40:22.737Z] skipping: [an-a01n01] [2026-04-16T06:40:22.737Z] skipping: [an-a01n02] [2026-04-16T06:40:22.737Z] [2026-04-16T06:40:22.737Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T06:40:22.737Z] skipping: [an-a01n01] [2026-04-16T06:40:22.737Z] skipping: [an-a01n02] [2026-04-16T06:40:22.737Z] [2026-04-16T06:40:22.737Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T06:40:22.737Z] skipping: [an-a01n01] [2026-04-16T06:40:22.737Z] skipping: [an-a01n02] [2026-04-16T06:40:22.737Z] [2026-04-16T06:40:22.737Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T06:40:22.737Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:22.737Z] [2026-04-16T06:40:22.737Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:40:22.737Z] ] *** [2026-04-16T06:40:24.577Z] skipping: [an-a01n01] [2026-04-16T06:40:24.577Z] ok: [an-a01n02] [2026-04-16T06:40:24.577Z] [2026-04-16T06:40:24.577Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T06:40:24.577Z] --ci-test \ [2026-04-16T06:40:24.577Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T06:40:24.577Z] --os almalinux9 \ [2026-04-16T06:40:24.577Z] --cpu 2 --ram 4G \ [2026-04-16T06:40:24.577Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T06:40:24.577Z] --install-media {{ installisouuid }} \ [2026-04-16T06:40:24.577Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T06:40:24.577Z] ] *** [2026-04-16T06:40:24.577Z] skipping: [an-a01n01] [2026-04-16T06:40:24.577Z] skipping: [an-a01n02] [2026-04-16T06:40:24.577Z] [2026-04-16T06:40:24.577Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T06:40:24.577Z] skipping: [an-a01n01] [2026-04-16T06:40:24.577Z] skipping: [an-a01n02] [2026-04-16T06:40:24.577Z] [2026-04-16T06:40:24.577Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:40:24.577Z] skipping: [an-a01n01] [2026-04-16T06:40:24.577Z] skipping: [an-a01n02] [2026-04-16T06:40:24.577Z] [2026-04-16T06:40:24.577Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T06:40:24.577Z] skipping: [an-a01n01] [2026-04-16T06:40:24.577Z] skipping: [an-a01n02] [2026-04-16T06:40:24.577Z] [2026-04-16T06:40:24.577Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T06:40:24.952Z] skipping: [an-a01n01] [2026-04-16T06:40:24.952Z] skipping: [an-a01n02] [2026-04-16T06:40:24.952Z] [2026-04-16T06:40:24.952Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T06:40:24.952Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:24.952Z] [2026-04-16T06:40:24.952Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:40:24.952Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:40:24.952Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:40:24.952Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:40:24.952Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:40:24.952Z] echo "pcs:" [2026-04-16T06:40:24.952Z] echo "$pcsstatus" [2026-04-16T06:40:24.952Z] exit 1 [2026-04-16T06:40:24.952Z] fi [2026-04-16T06:40:24.952Z] exit 0 [2026-04-16T06:40:24.952Z] ] *** [2026-04-16T06:40:27.435Z] ok: [an-a01n01] [2026-04-16T06:40:27.435Z] ok: [an-a01n02] [2026-04-16T06:40:27.435Z] [2026-04-16T06:40:27.435Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:40:27.435Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:40:27.435Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:40:27.435Z] ret=0 [2026-04-16T06:40:27.435Z] if [ "$servers" != "1" ]; then [2026-04-16T06:40:27.435Z] echo "Not all servers are running on the same node!" [2026-04-16T06:40:27.435Z] echo "$pcsstatus" [2026-04-16T06:40:27.435Z] ret=1 [2026-04-16T06:40:27.435Z] fi [2026-04-16T06:40:27.435Z] echo "$hypervisor" [2026-04-16T06:40:27.435Z] exit $ret [2026-04-16T06:40:27.435Z] ] *** [2026-04-16T06:40:29.275Z] ok: [an-a01n02] [2026-04-16T06:40:29.275Z] ok: [an-a01n01] [2026-04-16T06:40:29.275Z] [2026-04-16T06:40:29.275Z] PLAY [delete extra server tests] *********************************************** [2026-04-16T06:40:29.275Z] [2026-04-16T06:40:29.275Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T06:40:31.756Z] ok: [an-a01n01] [2026-04-16T06:40:31.756Z] ok: [an-a01n02] [2026-04-16T06:40:31.756Z] [2026-04-16T06:40:31.756Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T06:40:31.756Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:40:31.756Z] [2026-04-16T06:40:31.756Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:40:31.756Z] ] *** [2026-04-16T06:40:35.851Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:40:35.851Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:40:35.851Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:40:35.851Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:40:35.851Z] [2026-04-16T06:40:35.851Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-04-16T06:40:35.851Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T06:40:35.851Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T06:40:35.851Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T06:40:35.851Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T06:40:35.851Z] [2026-04-16T06:40:35.851Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:40:35.851Z] ] *** [2026-04-16T06:40:37.134Z] skipping: [an-a01n01] [2026-04-16T06:40:37.134Z] ok: [an-a01n02] [2026-04-16T06:40:37.134Z] [2026-04-16T06:40:37.134Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T06:40:37.134Z] --force \ [2026-04-16T06:40:37.134Z] --server an-test-deploy{{ servernum }} [2026-04-16T06:40:37.134Z] ] *** [2026-04-16T06:40:44.976Z] skipping: [an-a01n01] [2026-04-16T06:40:44.976Z] changed: [an-a01n02] [2026-04-16T06:40:44.976Z] [2026-04-16T06:40:44.976Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:40:44.976Z] ] *** [2026-04-16T06:41:13.484Z] skipping: [an-a01n01] [2026-04-16T06:41:13.484Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-04-16T06:41:13.484Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-04-16T06:41:13.484Z] ok: [an-a01n02] [2026-04-16T06:41:13.484Z] [2026-04-16T06:41:13.484Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:41:13.484Z] ] *** [2026-04-16T06:41:13.484Z] skipping: [an-a01n01] [2026-04-16T06:41:13.484Z] ok: [an-a01n02] [2026-04-16T06:41:13.484Z] [2026-04-16T06:41:13.484Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T06:41:13.484Z] ] *** [2026-04-16T06:41:13.484Z] skipping: [an-a01n01] [2026-04-16T06:41:13.484Z] ok: [an-a01n02] [2026-04-16T06:41:13.484Z] [2026-04-16T06:41:13.484Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:41:13.484Z] ] *** [2026-04-16T06:41:13.484Z] skipping: [an-a01n01] [2026-04-16T06:41:13.484Z] ok: [an-a01n02] [2026-04-16T06:41:13.484Z] [2026-04-16T06:41:13.484Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T06:41:13.484Z] --force \ [2026-04-16T06:41:13.484Z] --server an-test-deploy{{ servernum }} [2026-04-16T06:41:13.484Z] ] *** [2026-04-16T06:41:21.204Z] skipping: [an-a01n01] [2026-04-16T06:41:21.205Z] changed: [an-a01n02] [2026-04-16T06:41:21.205Z] [2026-04-16T06:41:21.205Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:41:21.205Z] ] *** [2026-04-16T06:42:45.300Z] skipping: [an-a01n01] [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-04-16T06:42:45.300Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-04-16T06:42:45.300Z] ok: [an-a01n02] [2026-04-16T06:42:45.300Z] [2026-04-16T06:42:45.300Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:42:45.300Z] ] *** [2026-04-16T06:42:45.300Z] skipping: [an-a01n01] [2026-04-16T06:42:45.300Z] ok: [an-a01n02] [2026-04-16T06:42:45.300Z] [2026-04-16T06:42:45.300Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T06:42:45.300Z] ] *** [2026-04-16T06:42:45.300Z] skipping: [an-a01n01] [2026-04-16T06:42:45.300Z] ok: [an-a01n02] [2026-04-16T06:42:45.300Z] [2026-04-16T06:42:45.300Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:42:45.300Z] ] *** [2026-04-16T06:42:45.300Z] skipping: [an-a01n01] [2026-04-16T06:42:45.300Z] ok: [an-a01n02] [2026-04-16T06:42:45.300Z] [2026-04-16T06:42:45.300Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T06:42:45.300Z] --force \ [2026-04-16T06:42:45.300Z] --server an-test-deploy{{ servernum }} [2026-04-16T06:42:45.300Z] ] *** [2026-04-16T06:42:45.300Z] skipping: [an-a01n01] [2026-04-16T06:42:45.300Z] changed: [an-a01n02] [2026-04-16T06:42:45.300Z] [2026-04-16T06:42:45.300Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:42:45.300Z] ] *** [2026-04-16T06:43:44.109Z] skipping: [an-a01n01] [2026-04-16T06:43:44.109Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-04-16T06:43:44.109Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-04-16T06:43:44.109Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-04-16T06:43:44.109Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-04-16T06:43:44.109Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-04-16T06:43:44.109Z] ok: [an-a01n02] [2026-04-16T06:43:44.109Z] [2026-04-16T06:43:44.109Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:43:44.109Z] ] *** [2026-04-16T06:43:44.109Z] skipping: [an-a01n01] [2026-04-16T06:43:44.109Z] ok: [an-a01n02] [2026-04-16T06:43:44.109Z] [2026-04-16T06:43:44.109Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T06:43:44.109Z] ] *** [2026-04-16T06:43:44.484Z] skipping: [an-a01n01] [2026-04-16T06:43:44.484Z] ok: [an-a01n02] [2026-04-16T06:43:44.484Z] [2026-04-16T06:43:44.484Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:43:44.484Z] ] *** [2026-04-16T06:43:46.324Z] skipping: [an-a01n01] [2026-04-16T06:43:46.324Z] ok: [an-a01n02] [2026-04-16T06:43:46.324Z] [2026-04-16T06:43:46.324Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T06:43:46.324Z] --force \ [2026-04-16T06:43:46.324Z] --server an-test-deploy{{ servernum }} [2026-04-16T06:43:46.324Z] ] *** [2026-04-16T06:43:54.045Z] skipping: [an-a01n01] [2026-04-16T06:43:54.045Z] changed: [an-a01n02] [2026-04-16T06:43:54.045Z] [2026-04-16T06:43:54.045Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:43:54.045Z] ] *** [2026-04-16T06:45:18.142Z] skipping: [an-a01n01] [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-04-16T06:45:18.142Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (55 retries left). [2026-04-16T06:45:18.142Z] ok: [an-a01n02] [2026-04-16T06:45:18.142Z] [2026-04-16T06:45:18.143Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:45:18.143Z] ] *** [2026-04-16T06:45:18.143Z] skipping: [an-a01n01] [2026-04-16T06:45:18.143Z] ok: [an-a01n02] [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T06:45:18.143Z] ] *** [2026-04-16T06:45:18.143Z] skipping: [an-a01n01] [2026-04-16T06:45:18.143Z] ok: [an-a01n02] [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] PLAY [create server for DR testing] ******************************************** [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T06:45:18.143Z] ok: [an-a01n02] [2026-04-16T06:45:18.143Z] ok: [an-a01n01] [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T06:45:18.143Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T06:45:18.143Z] ] *** [2026-04-16T06:45:18.143Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T06:45:18.143Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T06:45:18.143Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T06:45:18.143Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T06:45:18.143Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T06:45:18.143Z] [2026-04-16T06:45:18.143Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T06:45:18.143Z] if [ -z "$storage" ]; then [2026-04-16T06:45:18.143Z] echo "Unable to detect shared storage" [2026-04-16T06:45:18.143Z] exit 1 [2026-04-16T06:45:18.143Z] fi [2026-04-16T06:45:18.143Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T06:45:18.143Z] ] *** [2026-04-16T06:45:18.941Z] changed: [an-a01n01] [2026-04-16T06:45:18.941Z] changed: [an-a01n02] [2026-04-16T06:45:18.941Z] [2026-04-16T06:45:18.941Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T06:45:18.941Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T06:45:18.941Z] [2026-04-16T06:45:18.941Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T06:45:18.941Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T06:45:18.941Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T06:45:18.941Z] echo "Unable to detect all deploy iso" [2026-04-16T06:45:18.941Z] echo "Detected isos" [2026-04-16T06:45:18.941Z] echo "$isos" [2026-04-16T06:45:18.941Z] exit 1 [2026-04-16T06:45:18.941Z] fi [2026-04-16T06:45:18.941Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T06:45:18.941Z] if [ -z "$installiso" ]; then [2026-04-16T06:45:18.941Z] echo "Unable to detect install iso" [2026-04-16T06:45:18.941Z] echo "Detected isos" [2026-04-16T06:45:18.941Z] echo "$isos" [2026-04-16T06:45:18.941Z] exit 1 [2026-04-16T06:45:18.941Z] fi [2026-04-16T06:45:18.941Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T06:45:18.941Z] ] *** [2026-04-16T06:45:24.055Z] changed: [an-a01n01] [2026-04-16T06:45:24.055Z] changed: [an-a01n02] [2026-04-16T06:45:24.055Z] [2026-04-16T06:45:24.055Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T06:45:24.055Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T06:45:24.055Z] [2026-04-16T06:45:24.055Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T06:45:24.055Z] ] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] ok: [an-a01n01] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T06:45:25.894Z] --ci-test \ [2026-04-16T06:45:25.894Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T06:45:25.894Z] --os almalinux9 \ [2026-04-16T06:45:25.894Z] --cpu 2 --ram 4G \ [2026-04-16T06:45:25.894Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T06:45:25.894Z] --install-media {{ installisouuid }} \ [2026-04-16T06:45:25.894Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T06:45:25.894Z] ] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n01] [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n01] [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n01] [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n01] [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T06:45:25.894Z] skipping: [an-a01n01] [2026-04-16T06:45:25.894Z] skipping: [an-a01n02] [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T06:45:25.894Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T06:45:25.894Z] [2026-04-16T06:45:25.894Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:45:25.894Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T06:45:25.894Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T06:45:25.894Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T06:45:25.894Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T06:45:25.894Z] echo "pcs:" [2026-04-16T06:45:25.894Z] echo "$pcsstatus" [2026-04-16T06:45:25.894Z] exit 1 [2026-04-16T06:45:25.894Z] fi [2026-04-16T06:45:25.894Z] exit 0 [2026-04-16T06:45:25.894Z] ] *** [2026-04-16T06:45:28.376Z] ok: [an-a01n01] [2026-04-16T06:45:28.376Z] ok: [an-a01n02] [2026-04-16T06:45:28.376Z] [2026-04-16T06:45:28.376Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T06:45:28.376Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T06:45:28.376Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T06:45:28.376Z] ret=0 [2026-04-16T06:45:28.376Z] if [ "$servers" != "1" ]; then [2026-04-16T06:45:28.376Z] echo "Not all servers are running on the same node!" [2026-04-16T06:45:28.376Z] echo "$pcsstatus" [2026-04-16T06:45:28.376Z] ret=1 [2026-04-16T06:45:28.376Z] fi [2026-04-16T06:45:28.376Z] echo "$hypervisor" [2026-04-16T06:45:28.376Z] exit $ret [2026-04-16T06:45:28.376Z] ] *** [2026-04-16T06:45:30.214Z] ok: [an-a01n02] [2026-04-16T06:45:30.214Z] ok: [an-a01n01] [2026-04-16T06:45:30.214Z] [2026-04-16T06:45:30.214Z] PLAY [DR tests] **************************************************************** [2026-04-16T06:45:30.214Z] [2026-04-16T06:45:30.214Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T06:45:32.055Z] ok: [an-a01dr01] [2026-04-16T06:45:32.055Z] [2026-04-16T06:45:32.055Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured [2026-04-16T06:45:32.055Z] ] *** [2026-04-16T06:45:36.155Z] changed: [an-a01dr01] [2026-04-16T06:45:36.155Z] [2026-04-16T06:45:36.155Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g') [2026-04-16T06:45:36.155Z] echo "DB registered hostname: $drhostname" [2026-04-16T06:45:36.155Z] echo "current hostname: $(hostname)" [2026-04-16T06:45:36.155Z] if [ "$drhostname" != "$(hostname)" ]; then [2026-04-16T06:45:36.155Z] echo "DR hostnames are not matching" [2026-04-16T06:45:36.155Z] exit 1 [2026-04-16T06:45:36.155Z] fi [2026-04-16T06:45:36.155Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then [2026-04-16T06:45:36.155Z] echo "DR hostnames are not matching drnodes an-a01dr01" [2026-04-16T06:45:36.155Z] exit 1 [2026-04-16T06:45:36.155Z] fi [2026-04-16T06:45:36.155Z] ] *** [2026-04-16T06:45:40.252Z] changed: [an-a01dr01] [2026-04-16T06:45:40.252Z] [2026-04-16T06:45:40.252Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-04-16T06:45:40.252Z] echo "Detected a linked DR host!" [2026-04-16T06:45:40.252Z] exit 1 [2026-04-16T06:45:40.252Z] fi [2026-04-16T06:45:40.252Z] ] *** [2026-04-16T06:45:44.347Z] changed: [an-a01dr01] [2026-04-16T06:45:44.347Z] [2026-04-16T06:45:44.347Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01 [2026-04-16T06:45:44.347Z] ] *** [2026-04-16T06:45:48.443Z] changed: [an-a01dr01] [2026-04-16T06:45:48.443Z] [2026-04-16T06:45:48.443Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then [2026-04-16T06:45:48.443Z] echo "DR host failed to link to anvil" [2026-04-16T06:45:48.443Z] exit 1 [2026-04-16T06:45:48.443Z] fi [2026-04-16T06:45:48.443Z] ] *** [2026-04-16T06:45:52.572Z] changed: [an-a01dr01] [2026-04-16T06:45:52.572Z] [2026-04-16T06:45:52.572Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-04-16T06:45:52.572Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-04-16T06:45:52.572Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-04-16T06:45:52.572Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-04-16T06:45:52.572Z] exit 1 [2026-04-16T06:45:52.572Z] fi [2026-04-16T06:45:52.572Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then [2026-04-16T06:45:52.572Z] echo "Unable to link local vg uuid to storage group" [2026-04-16T06:45:52.572Z] exit 1 [2026-04-16T06:45:52.572Z] fi [2026-04-16T06:45:52.572Z] ] *** [2026-04-16T06:46:00.291Z] changed: [an-a01dr01] [2026-04-16T06:46:00.291Z] [2026-04-16T06:46:00.291Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then [2026-04-16T06:46:00.291Z] echo "Local vg does not appear to be linked to storage group" [2026-04-16T06:46:00.291Z] exit 1 [2026-04-16T06:46:00.291Z] fi [2026-04-16T06:46:00.291Z] ] *** [2026-04-16T06:46:02.771Z] changed: [an-a01dr01] [2026-04-16T06:46:02.771Z] [2026-04-16T06:46:02.771Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then [2026-04-16T06:46:02.771Z] echo "Unable to add protection to server an-test-deploy1" [2026-04-16T06:46:02.771Z] exit 1 [2026-04-16T06:46:02.771Z] fi [2026-04-16T06:46:02.771Z] ] *** [2026-04-16T06:46:07.886Z] changed: [an-a01dr01] [2026-04-16T06:46:07.886Z] [2026-04-16T06:46:07.886Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-04-16T06:46:07.886Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-16T06:46:07.886Z] [2026-04-16T06:46:07.886Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T06:46:07.886Z] exit 1 [2026-04-16T06:46:07.886Z] fi [2026-04-16T06:46:07.886Z] ] *** [2026-04-16T06:46:56.873Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-16T06:46:56.873Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-16T06:46:56.873Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-16T06:46:56.873Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left). [2026-04-16T06:46:56.873Z] ok: [an-a01dr01] [2026-04-16T06:46:56.873Z] [2026-04-16T06:46:56.873Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-04-16T06:46:56.873Z] short-throw) [2026-04-16T06:46:56.873Z] expected=2 [2026-04-16T06:46:56.873Z] protocol=A [2026-04-16T06:46:56.873Z] ;; [2026-04-16T06:46:56.873Z] sync) [2026-04-16T06:46:56.873Z] expected=3 [2026-04-16T06:46:56.873Z] protocol=C [2026-04-16T06:46:56.873Z] ;; [2026-04-16T06:46:56.873Z] *) [2026-04-16T06:46:56.873Z] echo "unknown DRBD protection method" [2026-04-16T06:46:56.873Z] exit 1 [2026-04-16T06:46:56.873Z] ;; [2026-04-16T06:46:56.873Z] esac [2026-04-16T06:46:56.873Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-04-16T06:46:56.873Z] echo "Unable to confirm correct DRBD protocol" [2026-04-16T06:46:56.873Z] exit 1 [2026-04-16T06:46:56.873Z] fi [2026-04-16T06:46:56.873Z] ] *** [2026-04-16T06:46:56.873Z] changed: [an-a01dr01] [2026-04-16T06:46:56.873Z] [2026-04-16T06:46:56.873Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:46:56.873Z] ] *** [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-16T06:49:06.492Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-04-16T06:49:06.492Z] ok: [an-a01dr01] [2026-04-16T06:49:06.492Z] [2026-04-16T06:49:06.492Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-04-16T06:49:06.492Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-16T06:49:06.492Z] [2026-04-16T06:49:06.492Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:49:06.492Z] ] *** [2026-04-16T06:49:06.492Z] ok: [an-a01dr01] [2026-04-16T06:49:06.492Z] [2026-04-16T06:49:06.492Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-04-16T06:49:06.492Z] echo "Unable to disconnect server from DR replication" [2026-04-16T06:49:06.492Z] exit 1 [2026-04-16T06:49:06.492Z] fi [2026-04-16T06:49:06.492Z] ] *** [2026-04-16T06:49:06.492Z] changed: [an-a01dr01] [2026-04-16T06:49:06.492Z] [2026-04-16T06:49:06.492Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:49:06.492Z] ] *** [2026-04-16T06:49:30.271Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T06:49:30.271Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T06:49:30.271Z] ok: [an-a01dr01] [2026-04-16T06:49:30.271Z] [2026-04-16T06:49:30.271Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-04-16T06:49:30.271Z] sleep 120 [2026-04-16T06:49:30.271Z] ] *** [2026-04-16T06:51:39.744Z] changed: [an-a01dr01 -> localhost] [2026-04-16T06:51:39.744Z] [2026-04-16T06:51:39.744Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-04-16T06:51:39.744Z] echo "Unable to update server DR replications" [2026-04-16T06:51:39.744Z] exit 1 [2026-04-16T06:51:39.744Z] fi [2026-04-16T06:51:39.744Z] ] *** [2026-04-16T06:51:39.744Z] changed: [an-a01dr01] [2026-04-16T06:51:39.744Z] [2026-04-16T06:51:39.744Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:51:39.744Z] ] *** [2026-04-16T06:52:03.530Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T06:52:03.530Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T06:52:03.530Z] ok: [an-a01dr01] [2026-04-16T06:52:03.530Z] [2026-04-16T06:52:03.530Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:52:03.530Z] ] *** [2026-04-16T06:52:13.039Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T06:52:13.039Z] ok: [an-a01dr01] [2026-04-16T06:52:13.039Z] [2026-04-16T06:52:13.039Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-04-16T06:52:13.039Z] echo "Unable to (re)connect server from DR replication" [2026-04-16T06:52:13.039Z] exit 1 [2026-04-16T06:52:13.039Z] fi [2026-04-16T06:52:13.039Z] ] *** [2026-04-16T06:52:17.132Z] changed: [an-a01dr01] [2026-04-16T06:52:17.132Z] [2026-04-16T06:52:17.132Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:52:17.132Z] ] *** [2026-04-16T06:52:40.914Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T06:52:40.914Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T06:52:40.914Z] ok: [an-a01dr01] [2026-04-16T06:52:40.914Z] [2026-04-16T06:52:40.914Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-04-16T06:52:40.914Z] echo "Unable to remove protection from server an-test-deploy1" [2026-04-16T06:52:40.914Z] exit 1 [2026-04-16T06:52:40.914Z] fi [2026-04-16T06:52:40.914Z] ] *** [2026-04-16T06:52:43.398Z] changed: [an-a01dr01] [2026-04-16T06:52:43.398Z] [2026-04-16T06:52:43.398Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-04-16T06:52:43.398Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-16T06:52:43.398Z] [2026-04-16T06:52:43.398Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:52:43.398Z] ] *** [2026-04-16T06:53:17.701Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-16T06:53:17.701Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-16T06:53:17.701Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-16T06:53:17.701Z] ok: [an-a01dr01] [2026-04-16T06:53:17.701Z] [2026-04-16T06:53:17.701Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T06:53:17.701Z] ] *** [2026-04-16T06:53:17.701Z] ok: [an-a01dr01] [2026-04-16T06:53:17.701Z] [2026-04-16T06:53:17.701Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T06:53:17.701Z] exit 1 [2026-04-16T06:53:17.701Z] fi [2026-04-16T06:53:17.701Z] ] *** [2026-04-16T06:53:52.014Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-16T06:53:52.014Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-16T06:53:52.014Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-04-16T06:53:52.014Z] ok: [an-a01dr01] [2026-04-16T06:53:52.014Z] [2026-04-16T06:53:52.014Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-04-16T06:53:52.014Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-04-16T06:53:52.014Z] #exit 1 [2026-04-16T06:53:52.014Z] fi [2026-04-16T06:53:52.014Z] ] *** [2026-04-16T06:53:52.014Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-16T06:53:52.014Z] [2026-04-16T06:53:52.014Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then [2026-04-16T06:53:52.014Z] echo "Unable to add protection to server an-test-deploy1" [2026-04-16T06:53:52.014Z] exit 1 [2026-04-16T06:53:52.014Z] fi [2026-04-16T06:53:52.014Z] ] *** [2026-04-16T06:53:55.111Z] changed: [an-a01dr01] [2026-04-16T06:53:55.111Z] [2026-04-16T06:53:55.111Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-04-16T06:53:55.111Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-16T06:53:55.111Z] [2026-04-16T06:53:55.111Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T06:53:55.111Z] exit 1 [2026-04-16T06:53:55.111Z] fi [2026-04-16T06:53:55.111Z] ] *** [2026-04-16T06:54:29.277Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-16T06:54:29.277Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-16T06:54:29.277Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-16T06:54:29.277Z] ok: [an-a01dr01] [2026-04-16T06:54:29.277Z] [2026-04-16T06:54:29.277Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-04-16T06:54:29.277Z] short-throw) [2026-04-16T06:54:29.277Z] expected=2 [2026-04-16T06:54:29.277Z] protocol=A [2026-04-16T06:54:29.277Z] ;; [2026-04-16T06:54:29.277Z] sync) [2026-04-16T06:54:29.277Z] expected=3 [2026-04-16T06:54:29.277Z] protocol=C [2026-04-16T06:54:29.277Z] ;; [2026-04-16T06:54:29.277Z] *) [2026-04-16T06:54:29.277Z] echo "unknown DRBD protection method" [2026-04-16T06:54:29.277Z] exit 1 [2026-04-16T06:54:29.277Z] ;; [2026-04-16T06:54:29.277Z] esac [2026-04-16T06:54:29.277Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-04-16T06:54:29.277Z] echo "Unable to confirm correct DRBD protocol" [2026-04-16T06:54:29.277Z] exit 1 [2026-04-16T06:54:29.277Z] fi [2026-04-16T06:54:29.277Z] ] *** [2026-04-16T06:54:29.277Z] changed: [an-a01dr01] [2026-04-16T06:54:29.277Z] [2026-04-16T06:54:29.277Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:54:29.277Z] ] *** [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-04-16T06:56:54.358Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left). [2026-04-16T06:56:54.358Z] ok: [an-a01dr01] [2026-04-16T06:56:54.358Z] [2026-04-16T06:56:54.358Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-04-16T06:56:54.358Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-16T06:56:54.358Z] [2026-04-16T06:56:54.358Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T06:56:54.358Z] ] *** [2026-04-16T06:56:54.358Z] ok: [an-a01dr01] [2026-04-16T06:56:54.358Z] [2026-04-16T06:56:54.358Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-04-16T06:56:54.358Z] echo "Unable to disconnect server from DR replication" [2026-04-16T06:56:54.358Z] exit 1 [2026-04-16T06:56:54.358Z] fi [2026-04-16T06:56:54.358Z] ] *** [2026-04-16T06:56:54.358Z] changed: [an-a01dr01] [2026-04-16T06:56:54.358Z] [2026-04-16T06:56:54.358Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:56:54.358Z] ] *** [2026-04-16T06:57:14.185Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T06:57:14.185Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T06:57:14.185Z] ok: [an-a01dr01] [2026-04-16T06:57:14.185Z] [2026-04-16T06:57:14.185Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-04-16T06:57:14.185Z] sleep 120 [2026-04-16T06:57:14.185Z] ] *** [2026-04-16T06:59:39.826Z] changed: [an-a01dr01 -> localhost] [2026-04-16T06:59:39.826Z] [2026-04-16T06:59:39.826Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-04-16T06:59:39.826Z] echo "Unable to update server DR replications" [2026-04-16T06:59:39.826Z] exit 1 [2026-04-16T06:59:39.826Z] fi [2026-04-16T06:59:39.826Z] ] *** [2026-04-16T06:59:39.826Z] changed: [an-a01dr01] [2026-04-16T06:59:39.826Z] [2026-04-16T06:59:39.826Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T06:59:39.826Z] ] *** [2026-04-16T07:00:05.005Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T07:00:05.006Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T07:00:05.006Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-04-16T07:00:05.006Z] ok: [an-a01dr01] [2026-04-16T07:00:05.006Z] [2026-04-16T07:00:05.006Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:00:05.006Z] ] *** [2026-04-16T07:00:26.383Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T07:00:26.383Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T07:00:26.383Z] ok: [an-a01dr01] [2026-04-16T07:00:26.383Z] [2026-04-16T07:00:26.383Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-04-16T07:00:26.383Z] echo "Unable to (re)connect server from DR replication" [2026-04-16T07:00:26.383Z] exit 1 [2026-04-16T07:00:26.383Z] fi [2026-04-16T07:00:26.383Z] ] *** [2026-04-16T07:00:30.485Z] changed: [an-a01dr01] [2026-04-16T07:00:30.485Z] [2026-04-16T07:00:30.485Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:00:30.485Z] ] *** [2026-04-16T07:00:54.305Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T07:00:54.305Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T07:00:54.305Z] ok: [an-a01dr01] [2026-04-16T07:00:54.305Z] [2026-04-16T07:00:54.305Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-04-16T07:00:54.305Z] echo "Unable to remove protection from server an-test-deploy1" [2026-04-16T07:00:54.305Z] exit 1 [2026-04-16T07:00:54.305Z] fi [2026-04-16T07:00:54.305Z] ] *** [2026-04-16T07:00:57.583Z] changed: [an-a01dr01] [2026-04-16T07:00:57.583Z] [2026-04-16T07:00:57.583Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-04-16T07:00:57.583Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-16T07:00:57.583Z] [2026-04-16T07:00:57.583Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:00:57.583Z] ] *** [2026-04-16T07:01:31.962Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-16T07:01:31.962Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-16T07:01:31.962Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-16T07:01:31.962Z] ok: [an-a01dr01] [2026-04-16T07:01:31.962Z] [2026-04-16T07:01:31.962Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:01:31.962Z] ] *** [2026-04-16T07:01:31.962Z] ok: [an-a01dr01] [2026-04-16T07:01:31.962Z] [2026-04-16T07:01:31.962Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T07:01:31.962Z] exit 1 [2026-04-16T07:01:31.962Z] fi [2026-04-16T07:01:31.962Z] ] *** [2026-04-16T07:02:03.922Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-16T07:02:03.922Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-16T07:02:03.922Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-04-16T07:02:03.922Z] ok: [an-a01dr01] [2026-04-16T07:02:03.922Z] [2026-04-16T07:02:03.922Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-04-16T07:02:03.922Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-04-16T07:02:03.922Z] #exit 1 [2026-04-16T07:02:03.922Z] fi [2026-04-16T07:02:03.922Z] ] *** [2026-04-16T07:02:03.922Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-16T07:02:03.922Z] [2026-04-16T07:02:03.922Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-04-16T07:02:03.922Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-04-16T07:02:03.922Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-04-16T07:02:03.922Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-04-16T07:02:03.922Z] exit 1 [2026-04-16T07:02:03.922Z] fi [2026-04-16T07:02:03.922Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then [2026-04-16T07:02:03.922Z] echo "Unable to unlink local vg uuid to storage group" [2026-04-16T07:02:03.922Z] exit 1 [2026-04-16T07:02:03.922Z] fi [2026-04-16T07:02:03.922Z] ] *** [2026-04-16T07:02:10.254Z] changed: [an-a01dr01] [2026-04-16T07:02:10.254Z] [2026-04-16T07:02:10.254Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then [2026-04-16T07:02:10.254Z] echo "Local vg does not appear to be unlinked from storage group (#886)" [2026-04-16T07:02:10.254Z] exit 1 [2026-04-16T07:02:10.254Z] fi [2026-04-16T07:02:10.254Z] ] *** [2026-04-16T07:02:13.485Z] changed: [an-a01dr01] [2026-04-16T07:02:13.485Z] [2026-04-16T07:02:13.485Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01 [2026-04-16T07:02:13.485Z] ] *** [2026-04-16T07:02:18.607Z] changed: [an-a01dr01] [2026-04-16T07:02:18.607Z] [2026-04-16T07:02:18.607Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-04-16T07:02:18.607Z] echo "Detected a linked DR host!" [2026-04-16T07:02:18.607Z] exit 1 [2026-04-16T07:02:18.607Z] fi [2026-04-16T07:02:18.607Z] ] *** [2026-04-16T07:02:22.451Z] changed: [an-a01dr01] [2026-04-16T07:02:22.451Z] [2026-04-16T07:02:22.451Z] PLAY [upgrade server test] ***************************************************** [2026-04-16T07:02:22.451Z] [2026-04-16T07:02:22.451Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:02:24.298Z] ok: [an-a01n01] [2026-04-16T07:02:24.298Z] ok: [an-a01n02] [2026-04-16T07:02:24.298Z] [2026-04-16T07:02:24.298Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T07:02:24.673Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T07:02:24.673Z] [2026-04-16T07:02:24.673Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T07:02:24.673Z] ] *** [2026-04-16T07:02:28.823Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T07:02:28.823Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T07:02:28.823Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T07:02:28.823Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T07:02:28.823Z] [2026-04-16T07:02:28.823Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T07:02:28.823Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T07:02:28.823Z] [2026-04-16T07:02:28.823Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T07:02:28.823Z] if [ -z "$storage" ]; then [2026-04-16T07:02:28.823Z] echo "Unable to detect shared storage" [2026-04-16T07:02:28.823Z] exit 1 [2026-04-16T07:02:28.823Z] fi [2026-04-16T07:02:28.823Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T07:02:28.823Z] ] *** [2026-04-16T07:02:33.948Z] changed: [an-a01n02] [2026-04-16T07:02:33.948Z] changed: [an-a01n01] [2026-04-16T07:02:33.948Z] [2026-04-16T07:02:33.948Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T07:02:33.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T07:02:33.948Z] [2026-04-16T07:02:33.948Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T07:02:33.948Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T07:02:33.948Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T07:02:33.948Z] echo "Unable to detect all deploy iso" [2026-04-16T07:02:33.948Z] echo "Detected isos" [2026-04-16T07:02:33.948Z] echo "$isos" [2026-04-16T07:02:33.948Z] exit 1 [2026-04-16T07:02:33.948Z] fi [2026-04-16T07:02:33.948Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T07:02:33.948Z] if [ -z "$installiso" ]; then [2026-04-16T07:02:33.948Z] echo "Unable to detect install iso" [2026-04-16T07:02:33.948Z] echo "Detected isos" [2026-04-16T07:02:33.948Z] echo "$isos" [2026-04-16T07:02:33.948Z] exit 1 [2026-04-16T07:02:33.948Z] fi [2026-04-16T07:02:33.948Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T07:02:33.948Z] ] *** [2026-04-16T07:02:39.229Z] changed: [an-a01n02] [2026-04-16T07:02:39.229Z] changed: [an-a01n01] [2026-04-16T07:02:39.229Z] [2026-04-16T07:02:39.229Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T07:02:39.229Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T07:02:39.229Z] [2026-04-16T07:02:39.229Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:02:39.229Z] ] *** [2026-04-16T07:02:41.081Z] skipping: [an-a01n02] [2026-04-16T07:02:41.081Z] ok: [an-a01n01] [2026-04-16T07:02:41.081Z] [2026-04-16T07:02:41.081Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 3fb60771-3d7b-40b4-bb70-cce37b320927 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T07:02:41.081Z] --ci-test \ [2026-04-16T07:02:41.081Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T07:02:41.081Z] --os almalinux9 \ [2026-04-16T07:02:41.081Z] --cpu 2 --ram 4G \ [2026-04-16T07:02:41.081Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T07:02:41.081Z] --install-media {{ installisouuid }} \ [2026-04-16T07:02:41.081Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T07:02:41.081Z] ] *** [2026-04-16T07:02:41.081Z] skipping: [an-a01n01] [2026-04-16T07:02:41.081Z] skipping: [an-a01n02] [2026-04-16T07:02:41.081Z] [2026-04-16T07:02:41.081Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T07:02:41.081Z] skipping: [an-a01n01] [2026-04-16T07:02:41.081Z] skipping: [an-a01n02] [2026-04-16T07:02:41.081Z] [2026-04-16T07:02:41.081Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T07:02:41.081Z] skipping: [an-a01n01] [2026-04-16T07:02:41.081Z] skipping: [an-a01n02] [2026-04-16T07:02:41.081Z] [2026-04-16T07:02:41.081Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T07:02:41.456Z] skipping: [an-a01n01] [2026-04-16T07:02:41.456Z] skipping: [an-a01n02] [2026-04-16T07:02:41.456Z] [2026-04-16T07:02:41.456Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T07:02:41.456Z] skipping: [an-a01n01] [2026-04-16T07:02:41.456Z] skipping: [an-a01n02] [2026-04-16T07:02:41.456Z] [2026-04-16T07:02:41.456Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T07:02:41.456Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T07:02:41.456Z] [2026-04-16T07:02:41.456Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T07:02:41.456Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T07:02:41.456Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T07:02:41.456Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T07:02:41.456Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T07:02:41.456Z] echo "pcs:" [2026-04-16T07:02:41.456Z] echo "$pcsstatus" [2026-04-16T07:02:41.456Z] exit 1 [2026-04-16T07:02:41.456Z] fi [2026-04-16T07:02:41.456Z] exit 0 [2026-04-16T07:02:41.456Z] ] *** [2026-04-16T07:02:43.770Z] ok: [an-a01n01] [2026-04-16T07:02:43.770Z] ok: [an-a01n02] [2026-04-16T07:02:43.770Z] [2026-04-16T07:02:43.770Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T07:02:43.770Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T07:02:43.770Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T07:02:43.770Z] ret=0 [2026-04-16T07:02:43.770Z] if [ "$servers" != "1" ]; then [2026-04-16T07:02:43.770Z] echo "Not all servers are running on the same node!" [2026-04-16T07:02:43.770Z] echo "$pcsstatus" [2026-04-16T07:02:43.770Z] ret=1 [2026-04-16T07:02:43.770Z] fi [2026-04-16T07:02:43.770Z] echo "$hypervisor" [2026-04-16T07:02:43.770Z] exit $ret [2026-04-16T07:02:43.770Z] ] *** [2026-04-16T07:02:45.615Z] ok: [an-a01n01] [2026-04-16T07:02:45.615Z] ok: [an-a01n02] [2026-04-16T07:02:45.615Z] [2026-04-16T07:02:45.615Z] PLAY [enable upgrade repo] ***************************************************** [2026-04-16T07:02:45.615Z] [2026-04-16T07:02:45.615Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:02:49.726Z] ok: [an-a01n01] [2026-04-16T07:02:49.726Z] ok: [an-a01dr01] [2026-04-16T07:02:49.726Z] ok: [an-a01n02] [2026-04-16T07:02:49.726Z] ok: [an-striker02] [2026-04-16T07:02:49.726Z] ok: [an-striker01] [2026-04-16T07:02:49.726Z] [2026-04-16T07:02:49.726Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-04-16T07:02:49.726Z] dnf config-manager --set-enabled $updaterepo [2026-04-16T07:02:49.726Z] ] *** [2026-04-16T07:02:51.119Z] changed: [an-a01n01] [2026-04-16T07:02:51.119Z] changed: [an-a01dr01] [2026-04-16T07:02:51.119Z] changed: [an-a01n02] [2026-04-16T07:02:51.119Z] changed: [an-striker02] [2026-04-16T07:02:51.119Z] changed: [an-striker01] [2026-04-16T07:02:51.119Z] [2026-04-16T07:02:51.119Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-04-16T07:02:51.119Z] ok: [an-striker01] [2026-04-16T07:02:51.496Z] ok: [an-striker02] [2026-04-16T07:02:51.496Z] ok: [an-a01n01] [2026-04-16T07:02:51.496Z] ok: [an-a01n02] [2026-04-16T07:02:51.496Z] ok: [an-a01dr01] [2026-04-16T07:02:51.496Z] [2026-04-16T07:02:51.496Z] PLAY [start upgrade process] *************************************************** [2026-04-16T07:02:51.496Z] [2026-04-16T07:02:51.496Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:02:53.339Z] ok: [an-striker01] [2026-04-16T07:02:53.339Z] [2026-04-16T07:02:53.339Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-04-16T07:02:53.339Z] ] *** [2026-04-16T07:25:30.055Z] changed: [an-striker01] [2026-04-16T07:25:30.055Z] [2026-04-16T07:25:30.055Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-04-16T07:25:53.907Z] changed: [an-striker01] [2026-04-16T07:25:53.907Z] [2026-04-16T07:25:53.907Z] PLAY [reconnect all nodes] ***************************************************** [2026-04-16T07:25:53.907Z] [2026-04-16T07:25:53.907Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:25:56.396Z] ok: [an-a01dr01] [2026-04-16T07:25:56.396Z] ok: [an-a01n02] [2026-04-16T07:25:56.396Z] ok: [an-a01n01] [2026-04-16T07:25:56.396Z] ok: [an-striker02] [2026-04-16T07:25:56.396Z] ok: [an-striker01] [2026-04-16T07:25:56.396Z] [2026-04-16T07:25:56.396Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-04-16T07:25:56.396Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01 [2026-04-16T07:25:56.396Z] [2026-04-16T07:25:56.396Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T07:29:07.015Z] ok: [an-striker01 -> localhost] [2026-04-16T07:29:07.015Z] ok: [an-a01n01 -> localhost] [2026-04-16T07:29:07.015Z] ok: [an-striker02 -> localhost] [2026-04-16T07:29:07.015Z] ok: [an-a01n02 -> localhost] [2026-04-16T07:29:07.015Z] ok: [an-a01dr01 -> localhost] [2026-04-16T07:29:07.015Z] [2026-04-16T07:29:07.015Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-04-16T07:29:07.015Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-04-16T07:29:07.015Z] exit 1 [2026-04-16T07:29:07.015Z] fi [2026-04-16T07:29:07.015Z] ] *** [2026-04-16T07:29:08.853Z] changed: [an-a01n01] [2026-04-16T07:29:08.853Z] changed: [an-a01n02] [2026-04-16T07:29:08.853Z] changed: [an-a01dr01] [2026-04-16T07:29:08.853Z] changed: [an-striker02] [2026-04-16T07:29:08.853Z] changed: [an-striker01] [2026-04-16T07:29:08.853Z] [2026-04-16T07:29:08.853Z] PLAY [check all nodes status after upgrade] ************************************ [2026-04-16T07:29:08.853Z] [2026-04-16T07:29:08.853Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:10.691Z] ok: [an-a01n01] [2026-04-16T07:29:10.691Z] ok: [an-striker02] [2026-04-16T07:29:10.691Z] ok: [an-a01dr01] [2026-04-16T07:29:10.691Z] ok: [an-a01n02] [2026-04-16T07:29:10.691Z] ok: [an-striker01] [2026-04-16T07:29:10.691Z] [2026-04-16T07:29:10.691Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-04-16T07:29:10.691Z] echo "perl-Log-Journald update not found!" [2026-04-16T07:29:10.691Z] exit 1 [2026-04-16T07:29:10.691Z] fi [2026-04-16T07:29:10.691Z] ] *** [2026-04-16T07:29:13.293Z] changed: [an-a01n01] [2026-04-16T07:29:13.293Z] changed: [an-a01dr01] [2026-04-16T07:29:13.293Z] changed: [an-a01n02] [2026-04-16T07:29:13.293Z] changed: [an-striker01] [2026-04-16T07:29:13.293Z] changed: [an-striker02] [2026-04-16T07:29:13.293Z] [2026-04-16T07:29:13.293Z] PLAY [check nodes status after upgrade] **************************************** [2026-04-16T07:29:13.293Z] [2026-04-16T07:29:13.293Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:15.131Z] ok: [an-a01dr01] [2026-04-16T07:29:15.131Z] ok: [an-a01n01] [2026-04-16T07:29:15.131Z] ok: [an-a01n02] [2026-04-16T07:29:15.131Z] [2026-04-16T07:29:15.131Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-04-16T07:29:15.131Z] echo "akmod-drbd update not found!" [2026-04-16T07:29:15.131Z] exit 1 [2026-04-16T07:29:15.131Z] fi [2026-04-16T07:29:15.131Z] ] *** [2026-04-16T07:29:16.414Z] changed: [an-a01n01] [2026-04-16T07:29:16.414Z] changed: [an-a01dr01] [2026-04-16T07:29:16.414Z] changed: [an-a01n02] [2026-04-16T07:29:16.414Z] [2026-04-16T07:29:16.414Z] PLAY [check hosted node status after upgrade] ********************************** [2026-04-16T07:29:16.414Z] [2026-04-16T07:29:16.414Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:18.252Z] ok: [an-a01n01] [2026-04-16T07:29:18.252Z] ok: [an-a01n02] [2026-04-16T07:29:18.252Z] [2026-04-16T07:29:18.252Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T07:29:18.252Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T07:29:18.252Z] [2026-04-16T07:29:18.252Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T07:29:18.252Z] ] *** [2026-04-16T07:29:22.347Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T07:29:22.347Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T07:29:22.347Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T07:29:22.347Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T07:29:22.347Z] [2026-04-16T07:29:22.347Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-04-16T07:29:22.347Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-04-16T07:29:22.347Z] [2026-04-16T07:29:22.347Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T07:29:22.347Z] ] *** [2026-04-16T07:29:23.630Z] ok: [an-a01n01] [2026-04-16T07:29:23.630Z] ok: [an-a01n02] [2026-04-16T07:29:23.630Z] [2026-04-16T07:29:23.630Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-04-16T07:29:23.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-04-16T07:29:23.630Z] [2026-04-16T07:29:23.630Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T07:29:23.630Z] ] *** [2026-04-16T07:29:24.531Z] ok: [an-a01n02] [2026-04-16T07:29:24.531Z] ok: [an-a01n01] [2026-04-16T07:29:24.531Z] [2026-04-16T07:29:24.531Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-04-16T07:29:24.531Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-04-16T07:29:24.531Z] [2026-04-16T07:29:24.531Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T07:29:24.531Z] ] *** [2026-04-16T07:29:25.329Z] ok: [an-a01n01] [2026-04-16T07:29:25.329Z] ok: [an-a01n02] [2026-04-16T07:29:25.329Z] [2026-04-16T07:29:25.329Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-04-16T07:29:25.329Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-04-16T07:29:25.329Z] [2026-04-16T07:29:25.329Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T07:29:25.329Z] ] *** [2026-04-16T07:29:26.610Z] ok: [an-a01n02] [2026-04-16T07:29:26.610Z] ok: [an-a01n01] [2026-04-16T07:29:26.610Z] [2026-04-16T07:29:26.610Z] PLAY [test anvil package self update] ****************************************** [2026-04-16T07:29:26.610Z] [2026-04-16T07:29:26.610Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:29.836Z] ok: [an-a01n01] [2026-04-16T07:29:29.836Z] ok: [an-a01n02] [2026-04-16T07:29:29.836Z] ok: [an-a01dr01] [2026-04-16T07:29:29.836Z] ok: [an-striker02] [2026-04-16T07:29:29.836Z] ok: [an-striker01] [2026-04-16T07:29:29.836Z] [2026-04-16T07:29:29.836Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-04-16T07:29:33.059Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-16T07:29:33.059Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-16T07:29:33.059Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-16T07:29:33.059Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-16T07:29:33.059Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-16T07:29:33.059Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-16T07:29:33.059Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-16T07:29:33.059Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-16T07:29:33.059Z] changed: [an-striker02] => (item=scancore.service) [2026-04-16T07:29:33.059Z] changed: [an-striker01] => (item=scancore.service) [2026-04-16T07:29:33.059Z] [2026-04-16T07:29:33.059Z] PLAY [test striker package self update] **************************************** [2026-04-16T07:29:33.059Z] [2026-04-16T07:29:33.059Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:35.088Z] ok: [an-striker01] [2026-04-16T07:29:35.088Z] ok: [an-striker02] [2026-04-16T07:29:35.088Z] [2026-04-16T07:29:35.088Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-04-16T07:29:35.088Z] ] *** [2026-04-16T07:29:51.585Z] changed: [an-striker02] [2026-04-16T07:29:51.585Z] changed: [an-striker01] [2026-04-16T07:29:51.585Z] [2026-04-16T07:29:51.585Z] PLAY [test node package self update] ******************************************* [2026-04-16T07:29:51.585Z] [2026-04-16T07:29:51.585Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:29:53.425Z] ok: [an-a01n02] [2026-04-16T07:29:53.425Z] ok: [an-a01n01] [2026-04-16T07:29:53.425Z] [2026-04-16T07:29:53.425Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-04-16T07:29:53.425Z] ] *** [2026-04-16T07:30:10.308Z] changed: [an-a01n01] [2026-04-16T07:30:10.308Z] changed: [an-a01n02] [2026-04-16T07:30:10.308Z] [2026-04-16T07:30:10.308Z] PLAY [test dr package self update] ********************************************* [2026-04-16T07:30:10.308Z] [2026-04-16T07:30:10.308Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:30:11.105Z] ok: [an-a01dr01] [2026-04-16T07:30:11.105Z] [2026-04-16T07:30:11.105Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr [2026-04-16T07:30:11.105Z] ] *** [2026-04-16T07:30:27.600Z] changed: [an-a01dr01] [2026-04-16T07:30:27.600Z] [2026-04-16T07:30:27.600Z] PLAY [test anvil package self update] ****************************************** [2026-04-16T07:30:27.600Z] [2026-04-16T07:30:27.600Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:30:28.398Z] ok: [an-striker02] [2026-04-16T07:30:28.398Z] ok: [an-striker01] [2026-04-16T07:30:28.398Z] ok: [an-a01n01] [2026-04-16T07:30:28.398Z] ok: [an-a01n02] [2026-04-16T07:30:28.398Z] ok: [an-a01dr01] [2026-04-16T07:30:28.398Z] [2026-04-16T07:30:28.398Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-04-16T07:30:30.881Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-16T07:30:30.881Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-16T07:30:30.881Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-16T07:30:30.881Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-16T07:30:30.881Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-16T07:30:30.881Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-16T07:30:30.881Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-16T07:30:30.881Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-16T07:30:30.881Z] changed: [an-striker02] => (item=scancore.service) [2026-04-16T07:30:30.881Z] changed: [an-striker01] => (item=scancore.service) [2026-04-16T07:30:30.881Z] [2026-04-16T07:30:30.881Z] PLAY [delete server tests] ***************************************************** [2026-04-16T07:30:30.881Z] [2026-04-16T07:30:30.881Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:30:32.720Z] ok: [an-a01n01] [2026-04-16T07:30:32.720Z] ok: [an-a01n02] [2026-04-16T07:30:32.720Z] [2026-04-16T07:30:32.720Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T07:30:32.720Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T07:30:32.720Z] [2026-04-16T07:30:32.720Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T07:30:32.720Z] ] *** [2026-04-16T07:30:36.816Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T07:30:36.816Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T07:30:36.816Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T07:30:36.816Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T07:30:36.816Z] [2026-04-16T07:30:36.816Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-04-16T07:30:37.190Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T07:30:37.190Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T07:30:37.190Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T07:30:37.190Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T07:30:37.190Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T07:30:37.190Z] [2026-04-16T07:30:37.190Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:30:37.190Z] ] *** [2026-04-16T07:30:38.833Z] skipping: [an-a01n01] [2026-04-16T07:30:38.833Z] ok: [an-a01n02] [2026-04-16T07:30:38.833Z] [2026-04-16T07:30:38.833Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T07:30:38.833Z] --force \ [2026-04-16T07:30:38.833Z] --server an-test-deploy{{ servernum }} [2026-04-16T07:30:38.833Z] ] *** [2026-04-16T07:30:48.223Z] skipping: [an-a01n01] [2026-04-16T07:30:48.223Z] changed: [an-a01n02] [2026-04-16T07:30:48.223Z] [2026-04-16T07:30:48.223Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:30:48.223Z] ] *** [2026-04-16T07:31:37.224Z] skipping: [an-a01n01] [2026-04-16T07:31:37.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-04-16T07:31:37.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-04-16T07:31:37.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-04-16T07:31:37.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-04-16T07:31:37.224Z] ok: [an-a01n02] [2026-04-16T07:31:37.224Z] [2026-04-16T07:31:37.224Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:31:37.224Z] ] *** [2026-04-16T07:31:37.224Z] skipping: [an-a01n01] [2026-04-16T07:31:37.224Z] ok: [an-a01n02] [2026-04-16T07:31:37.224Z] [2026-04-16T07:31:37.224Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:31:37.224Z] ] *** [2026-04-16T07:31:37.224Z] skipping: [an-a01n01] [2026-04-16T07:31:37.224Z] ok: [an-a01n02] [2026-04-16T07:31:37.224Z] [2026-04-16T07:31:37.224Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:37.224Z] ] *** [2026-04-16T07:31:38.321Z] skipping: [an-a01n01] [2026-04-16T07:31:38.321Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.916441", "end": "2026-04-16 07:31:37.577125", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 07:31:36.660684", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T07:31:38.321Z] ...ignoring [2026-04-16T07:31:38.321Z] [2026-04-16T07:31:38.321Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T07:31:38.321Z] --force \ [2026-04-16T07:31:38.321Z] --server an-test-deploy{{ servernum }} [2026-04-16T07:31:38.321Z] ] *** [2026-04-16T07:31:38.321Z] skipping: [an-a01n01] [2026-04-16T07:31:38.321Z] skipping: [an-a01n02] [2026-04-16T07:31:38.321Z] [2026-04-16T07:31:38.321Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:38.321Z] ] *** [2026-04-16T07:31:38.321Z] skipping: [an-a01n01] [2026-04-16T07:31:38.321Z] skipping: [an-a01n02] [2026-04-16T07:31:38.321Z] [2026-04-16T07:31:38.321Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:31:38.321Z] ] *** [2026-04-16T07:31:38.321Z] skipping: [an-a01n01] [2026-04-16T07:31:38.321Z] skipping: [an-a01n02] [2026-04-16T07:31:38.321Z] [2026-04-16T07:31:38.321Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:31:38.321Z] ] *** [2026-04-16T07:31:38.321Z] skipping: [an-a01n01] [2026-04-16T07:31:38.321Z] skipping: [an-a01n02] [2026-04-16T07:31:38.321Z] [2026-04-16T07:31:38.321Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:38.321Z] ] *** [2026-04-16T07:31:39.604Z] skipping: [an-a01n01] [2026-04-16T07:31:39.605Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.879787", "end": "2026-04-16 07:31:39.418061", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 07:31:38.538274", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T07:31:39.605Z] ...ignoring [2026-04-16T07:31:39.605Z] [2026-04-16T07:31:39.605Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T07:31:39.605Z] --force \ [2026-04-16T07:31:39.605Z] --server an-test-deploy{{ servernum }} [2026-04-16T07:31:39.605Z] ] *** [2026-04-16T07:31:39.605Z] skipping: [an-a01n01] [2026-04-16T07:31:39.605Z] skipping: [an-a01n02] [2026-04-16T07:31:39.605Z] [2026-04-16T07:31:39.605Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:39.605Z] ] *** [2026-04-16T07:31:39.979Z] skipping: [an-a01n01] [2026-04-16T07:31:39.979Z] skipping: [an-a01n02] [2026-04-16T07:31:39.979Z] [2026-04-16T07:31:39.979Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:31:39.979Z] ] *** [2026-04-16T07:31:39.979Z] skipping: [an-a01n01] [2026-04-16T07:31:39.979Z] skipping: [an-a01n02] [2026-04-16T07:31:39.979Z] [2026-04-16T07:31:39.979Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:31:39.979Z] ] *** [2026-04-16T07:31:39.979Z] skipping: [an-a01n01] [2026-04-16T07:31:39.979Z] skipping: [an-a01n02] [2026-04-16T07:31:39.979Z] [2026-04-16T07:31:39.979Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:39.979Z] ] *** [2026-04-16T07:31:41.819Z] skipping: [an-a01n01] [2026-04-16T07:31:41.819Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.934112", "end": "2026-04-16 07:31:41.235919", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 07:31:40.301807", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T07:31:41.819Z] ...ignoring [2026-04-16T07:31:41.819Z] [2026-04-16T07:31:41.819Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T07:31:41.819Z] --force \ [2026-04-16T07:31:41.819Z] --server an-test-deploy{{ servernum }} [2026-04-16T07:31:41.819Z] ] *** [2026-04-16T07:31:41.819Z] skipping: [an-a01n01] [2026-04-16T07:31:41.819Z] skipping: [an-a01n02] [2026-04-16T07:31:41.819Z] [2026-04-16T07:31:41.819Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:41.819Z] ] *** [2026-04-16T07:31:41.819Z] skipping: [an-a01n01] [2026-04-16T07:31:41.819Z] skipping: [an-a01n02] [2026-04-16T07:31:41.819Z] [2026-04-16T07:31:41.819Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:31:41.819Z] ] *** [2026-04-16T07:31:41.819Z] skipping: [an-a01n01] [2026-04-16T07:31:41.819Z] skipping: [an-a01n02] [2026-04-16T07:31:41.819Z] [2026-04-16T07:31:41.819Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:31:41.819Z] ] *** [2026-04-16T07:31:41.819Z] skipping: [an-a01n01] [2026-04-16T07:31:41.819Z] skipping: [an-a01n02] [2026-04-16T07:31:41.819Z] [2026-04-16T07:31:41.819Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:41.819Z] ] *** [2026-04-16T07:31:43.658Z] skipping: [an-a01n01] [2026-04-16T07:31:43.658Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.900172", "end": "2026-04-16 07:31:43.125011", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 07:31:42.224839", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T07:31:43.658Z] ...ignoring [2026-04-16T07:31:43.658Z] [2026-04-16T07:31:43.658Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T07:31:43.658Z] --force \ [2026-04-16T07:31:43.658Z] --server an-test-deploy{{ servernum }} [2026-04-16T07:31:43.658Z] ] *** [2026-04-16T07:31:43.658Z] skipping: [an-a01n01] [2026-04-16T07:31:43.658Z] skipping: [an-a01n02] [2026-04-16T07:31:43.658Z] [2026-04-16T07:31:43.658Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T07:31:43.658Z] ] *** [2026-04-16T07:31:43.658Z] skipping: [an-a01n01] [2026-04-16T07:31:43.658Z] skipping: [an-a01n02] [2026-04-16T07:31:43.658Z] [2026-04-16T07:31:43.658Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T07:31:43.658Z] ] *** [2026-04-16T07:31:43.658Z] skipping: [an-a01n01] [2026-04-16T07:31:43.658Z] skipping: [an-a01n02] [2026-04-16T07:31:43.658Z] [2026-04-16T07:31:43.658Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T07:31:43.658Z] ] *** [2026-04-16T07:31:43.658Z] skipping: [an-a01n01] [2026-04-16T07:31:43.658Z] skipping: [an-a01n02] [2026-04-16T07:31:43.658Z] [2026-04-16T07:31:43.659Z] PLAY [Check anvil available resources from striker01] ************************** [2026-04-16T07:31:43.659Z] [2026-04-16T07:31:43.659Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:31:46.140Z] ok: [an-striker01] [2026-04-16T07:31:46.140Z] [2026-04-16T07:31:46.140Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-04-16T07:31:46.140Z] sleep 300 [2026-04-16T07:31:46.140Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-04-16T07:31:46.140Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-04-16T07:31:46.140Z] ] *** [2026-04-16T07:36:57.208Z] ok: [an-striker01] [2026-04-16T07:36:57.208Z] [2026-04-16T07:36:57.208Z] PLAY RECAP ********************************************************************* [2026-04-16T07:36:57.208Z] an-a01dr01 : ok=90 changed=38 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [2026-04-16T07:36:57.208Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-04-16T07:36:57.208Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-04-16T07:36:57.208Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-16T07:36:57.208Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-16T07:36:57.208Z] [2026-04-16T07:36:57.208Z] tar: Removing leading `/' from member names [2026-04-16T07:36:57.208Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-16T07:36:57.208Z] /root/ci-tools [2026-04-16T07:36:57.208Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/341.tar.gz [2026-04-16T07:36:57.302Z] [Pipeline] } [2026-04-16T07:36:57.305Z] [Pipeline] // timeout [2026-04-16T07:36:57.309Z] [Pipeline] } [2026-04-16T07:36:57.312Z] [Pipeline] // script [2026-04-16T07:36:57.315Z] [Pipeline] } [2026-04-16T07:36:57.317Z] [Pipeline] // stage [2026-04-16T07:36:57.321Z] [Pipeline] stage [2026-04-16T07:36:57.322Z] [Pipeline] { (Declarative: Post Actions) [2026-04-16T07:36:57.329Z] [Pipeline] cleanWs [2026-04-16T07:36:57.394Z] [WS-CLEANUP] Deleting project workspace... [2026-04-16T07:36:57.394Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-16T07:36:57.426Z] [WS-CLEANUP] done [2026-04-16T07:36:57.429Z] [Pipeline] script [2026-04-16T07:36:57.430Z] [Pipeline] { [2026-04-16T07:36:57.452Z] [Pipeline] stage [2026-04-16T07:36:57.452Z] [Pipeline] { (Copying logs) [2026-04-16T07:36:57.483Z] [Pipeline] node [2026-04-16T07:36:57.490Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-04-16T07:36:57.490Z] [Pipeline] { [2026-04-16T07:36:57.524Z] [Pipeline] node [2026-04-16T07:36:57.531Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-04-16T07:36:57.531Z] [Pipeline] { [2026-04-16T07:36:57.535Z] [Pipeline] sh [2026-04-16T07:36:57.790Z] + mkdir -p /var/lib/jenkins/locks [2026-04-16T07:36:57.798Z] [Pipeline] echo [2026-04-16T07:36:57.798Z] RWLock: FD for lock log_archive in stage projectFinishUp is 468 [2026-04-16T07:36:57.801Z] [Pipeline] sleep [2026-04-16T07:36:57.801Z] No need to sleep any longer [2026-04-16T07:36:57.804Z] [Pipeline] echo [2026-04-16T07:36:57.804Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-04-16T07:36:57.807Z] [Pipeline] } [2026-04-16T07:36:57.809Z] [Pipeline] // node [2026-04-16T07:36:57.812Z] [Pipeline] echo [2026-04-16T07:36:57.812Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/341/ [2026-04-16T07:36:57.813Z] [Pipeline] sh [2026-04-16T07:36:58.068Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/341/ [2026-04-16T07:36:58.069Z] [Pipeline] sh