[2026-09-04T16:42:08.614Z] Started by upstream project "anvil/anvil-pipeline/PR-1002" build number 2 [2026-09-04T16:42:08.614Z] originally caused by: [2026-09-04T16:42:08.614Z] Pull request #1002 updated [2026-09-04T16:42:09.309Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-09-04T16:42:09.318Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-09-04T16:42:09.605Z] Examining kronosnet/ci-tools [2026-09-04T16:42:09.605Z] Attempting to resolve main as a branch [2026-09-04T16:42:09.767Z] Resolved main as branch main at revision 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 [2026-09-04T16:42:09.768Z] Selected Git installation does not exist. Using Default [2026-09-04T16:42:09.768Z] The recommended git tool is: NONE [2026-09-04T16:42:09.769Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-09-04T16:42:09.771Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-09-04T16:42:09.771Z] Wiping out workspace first. [2026-09-04T16:42:09.798Z] Cloning the remote Git repository [2026-09-04T16:42:09.798Z] Using shallow clone with depth 1 [2026-09-04T16:42:09.798Z] Avoid fetching tags [2026-09-04T16:42:09.798Z] Honoring refspec on initial clone [2026-09-04T16:42:09.798Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-09-04T16:42:09.798Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10 [2026-09-04T16:42:09.807Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-09-04T16:42:09.807Z] > git --version # timeout=10 [2026-09-04T16:42:09.815Z] > git --version # 'git version 2.52.0' [2026-09-04T16:42:09.815Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-09-04T16:42:09.822Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-09-04T16:42:10.176Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-09-04T16:42:10.183Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-09-04T16:42:10.192Z] Avoid second fetch [2026-09-04T16:42:10.192Z] Checking out Revision 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 (main) [2026-09-04T16:42:10.192Z] > git config core.sparsecheckout # timeout=10 [2026-09-04T16:42:10.200Z] > git checkout -f 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 # timeout=10 [2026-09-04T16:42:10.236Z] Commit message: "fn-testing: remove unsupported combination" [2026-09-04T16:42:10.245Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-09-04T16:42:10.505Z] Examining kronosnet/ci-tools [2026-09-04T16:42:10.505Z] Attempting to resolve main as a branch [2026-09-04T16:42:10.678Z] Resolved main as branch main at revision 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 [2026-09-04T16:42:10.679Z] Selected Git installation does not exist. Using Default [2026-09-04T16:42:10.679Z] The recommended git tool is: NONE [2026-09-04T16:42:10.680Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-09-04T16:42:10.684Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-09-04T16:42:10.684Z] Wiping out workspace first. [2026-09-04T16:42:10.705Z] Cloning the remote Git repository [2026-09-04T16:42:10.705Z] Using shallow clone with depth 1 [2026-09-04T16:42:10.705Z] Avoid fetching tags [2026-09-04T16:42:10.705Z] Honoring refspec on initial clone [2026-09-04T16:42:10.705Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-09-04T16:42:10.705Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-09-04T16:42:10.716Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-09-04T16:42:10.716Z] > git --version # timeout=10 [2026-09-04T16:42:10.723Z] > git --version # 'git version 2.52.0' [2026-09-04T16:42:10.723Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-09-04T16:42:10.731Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-09-04T16:42:11.083Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-09-04T16:42:11.091Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-09-04T16:42:11.099Z] Avoid second fetch [2026-09-04T16:42:11.099Z] Checking out Revision 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 (main) [2026-09-04T16:42:11.099Z] > git config core.sparsecheckout # timeout=10 [2026-09-04T16:42:11.107Z] > git checkout -f 0c6b49fadf2f7a5c1fcb42ab9b6ec22b46735863 # timeout=10 [2026-09-04T16:42:11.141Z] Commit message: "fn-testing: remove unsupported combination" [2026-09-04T16:42:11.246Z] [Pipeline] Start of Pipeline [2026-09-04T16:42:11.343Z] [Pipeline] node [2026-09-04T16:42:11.347Z] Running on anvil-ci-bm-phy07 in /srv/workspace/anvil/anvil-build-functional-testing [2026-09-04T16:42:11.347Z] [Pipeline] { [2026-09-04T16:42:11.357Z] [Pipeline] stage [2026-09-04T16:42:11.357Z] [Pipeline] { (Prep build env) [2026-09-04T16:42:11.377Z] [Pipeline] script [2026-09-04T16:42:11.379Z] [Pipeline] { [2026-09-04T16:42:11.383Z] [Pipeline] cleanWs [2026-09-04T16:42:11.433Z] [WS-CLEANUP] Deleting project workspace... [2026-09-04T16:42:11.433Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-09-04T16:42:11.461Z] [WS-CLEANUP] done [2026-09-04T16:42:11.485Z] [Pipeline] lock [2026-09-04T16:42:11.485Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-09-04T16:42:11.486Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-09-04T16:42:11.486Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-09-04T16:42:11.486Z] [Pipeline] { [2026-09-04T16:42:11.489Z] [Pipeline] } [2026-09-04T16:42:11.489Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-09-04T16:42:11.492Z] [Pipeline] // lock [2026-09-04T16:42:11.497Z] [Pipeline] } [2026-09-04T16:42:11.500Z] [Pipeline] // script [2026-09-04T16:42:11.503Z] [Pipeline] } [2026-09-04T16:42:11.505Z] [Pipeline] // stage [2026-09-04T16:42:11.508Z] [Pipeline] stage [2026-09-04T16:42:11.508Z] [Pipeline] { (Run functional testing) [2026-09-04T16:42:11.516Z] [Pipeline] script [2026-09-04T16:42:11.517Z] [Pipeline] { [2026-09-04T16:42:11.540Z] [Pipeline] echo [2026-09-04T16:42:11.540Z] runWithTimeout 360 [2026-09-04T16:42:11.541Z] [Pipeline] timeout [2026-09-04T16:42:11.541Z] Timeout set to expire in 6 hr 0 min [2026-09-04T16:42:11.541Z] [Pipeline] { [2026-09-04T16:42:11.548Z] [Pipeline] sh [2026-09-04T16:42:12.038Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 ci pr/1002 all [2026-09-04T16:42:12.038Z] CITBRANCH: main [2026-09-04T16:42:12.038Z] CITHOME: /root/ci-tools [2026-09-04T16:42:12.038Z] CITCMD: ci-setup-anvil-bm-vm [2026-09-04T16:42:12.038Z] CITCMDARGS: rhel-9 ci pr/1002 all [2026-09-04T16:42:12.381Z] Genering ssh keys and ssh config [2026-09-04T16:42:12.723Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-09-04T16:45:21.814Z] Destroying VMs (if any) [2026-09-04T16:45:21.814Z] Warning: Permanently added 'an-striker01' (ED25519) to the list of known hosts. [2026-09-04T16:45:21.814Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-09-04T16:45:21.814Z] This might take a few seconds. [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T16:45:21.814Z] ! Cannot disconnect from Red Hat Insights [2026-09-04T16:45:21.814Z] ! Cannot disconnect from Red Hat Subscription Management [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] Manage your connected systems: https://red.ht/connector [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] The following errors were encountered during disconnect: [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] TYPE STEP ERROR [2026-09-04T16:45:21.814Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-09-04T16:45:21.814Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-09-04T16:45:21.814Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] 1. Register with subscription-manager [2026-09-04T16:45:21.814Z] # subscription-manager register [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] 2. Register with insights-client [2026-09-04T16:45:21.814Z] # insights-client --register [2026-09-04T16:45:21.814Z] This system is currently not registered. [2026-09-04T16:45:21.814Z] Domain 'an-striker01' destroyed [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] Domain 'an-striker01' has been undefined [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] Warning: Permanently added 'an-striker02' (ED25519) to the list of known hosts. [2026-09-04T16:45:21.814Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-09-04T16:45:21.814Z] This might take a few seconds. [2026-09-04T16:45:21.814Z] [2026-09-04T16:45:21.814Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T16:45:22.999Z] ! Cannot disconnect from Red Hat Insights [2026-09-04T16:45:22.999Z] ! Cannot disconnect from Red Hat Subscription Management [2026-09-04T16:45:22.999Z] [2026-09-04T16:45:22.999Z] Manage your connected systems: https://red.ht/connector [2026-09-04T16:45:22.999Z] [2026-09-04T16:45:22.999Z] The following errors were encountered during disconnect: [2026-09-04T16:45:22.999Z] [2026-09-04T16:45:22.999Z] TYPE STEP ERROR [2026-09-04T16:45:22.999Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-09-04T16:45:22.999Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-09-04T16:45:24.708Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T16:45:24.708Z] [2026-09-04T16:45:24.708Z] 1. Register with subscription-manager [2026-09-04T16:45:24.708Z] # subscription-manager register [2026-09-04T16:45:24.708Z] [2026-09-04T16:45:24.708Z] 2. Register with insights-client [2026-09-04T16:45:24.708Z] # insights-client --register [2026-09-04T16:45:25.443Z] This system is currently not registered. [2026-09-04T16:45:27.762Z] Domain 'an-striker02' destroyed [2026-09-04T16:45:27.762Z] [2026-09-04T16:45:27.762Z] Domain 'an-striker02' has been undefined [2026-09-04T16:45:27.762Z] [2026-09-04T16:45:28.104Z] Warning: Permanently added 'an-a01n01' (ED25519) to the list of known hosts. [2026-09-04T16:45:28.104Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-09-04T16:45:28.104Z] This might take a few seconds. [2026-09-04T16:45:28.104Z] [2026-09-04T16:45:28.104Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T16:45:29.288Z] ! Cannot disconnect from Red Hat Insights [2026-09-04T16:45:29.288Z] ! Cannot disconnect from Red Hat Subscription Management [2026-09-04T16:45:29.288Z] [2026-09-04T16:45:29.288Z] Manage your connected systems: https://red.ht/connector [2026-09-04T16:45:29.288Z] [2026-09-04T16:45:29.288Z] The following errors were encountered during disconnect: [2026-09-04T16:45:29.288Z] [2026-09-04T16:45:29.288Z] TYPE STEP ERROR [2026-09-04T16:45:29.288Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-09-04T16:45:29.288Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-09-04T16:45:30.474Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T16:45:30.474Z] [2026-09-04T16:45:30.474Z] 1. Register with subscription-manager [2026-09-04T16:45:30.474Z] # subscription-manager register [2026-09-04T16:45:30.474Z] [2026-09-04T16:45:30.474Z] 2. Register with insights-client [2026-09-04T16:45:30.474Z] # insights-client --register [2026-09-04T16:45:31.660Z] This system is currently not registered. [2026-09-04T16:45:34.692Z] Domain 'an-a01n01' destroyed [2026-09-04T16:45:34.692Z] [2026-09-04T16:45:34.692Z] Domain 'an-a01n01' has been undefined [2026-09-04T16:45:34.692Z] [2026-09-04T16:45:34.692Z] Warning: Permanently added 'an-a01n02' (ED25519) to the list of known hosts. [2026-09-04T16:45:35.034Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-09-04T16:45:35.034Z] This might take a few seconds. [2026-09-04T16:45:35.034Z] [2026-09-04T16:45:35.034Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T16:45:36.218Z] ! Cannot disconnect from Red Hat Insights [2026-09-04T16:45:36.218Z] ! Cannot disconnect from Red Hat Subscription Management [2026-09-04T16:45:36.218Z] [2026-09-04T16:45:36.218Z] Manage your connected systems: https://red.ht/connector [2026-09-04T16:45:36.218Z] [2026-09-04T16:45:36.218Z] The following errors were encountered during disconnect: [2026-09-04T16:45:36.218Z] [2026-09-04T16:45:36.218Z] TYPE STEP ERROR [2026-09-04T16:45:36.218Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-09-04T16:45:36.218Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-09-04T16:45:37.404Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T16:45:37.404Z] [2026-09-04T16:45:37.404Z] 1. Register with subscription-manager [2026-09-04T16:45:37.404Z] # subscription-manager register [2026-09-04T16:45:37.404Z] [2026-09-04T16:45:37.404Z] 2. Register with insights-client [2026-09-04T16:45:37.404Z] # insights-client --register [2026-09-04T16:45:38.589Z] This system is currently not registered. [2026-09-04T16:45:40.907Z] Domain 'an-a01n02' destroyed [2026-09-04T16:45:40.907Z] [2026-09-04T16:45:40.907Z] Domain 'an-a01n02' has been undefined [2026-09-04T16:45:40.907Z] [2026-09-04T16:45:40.907Z] Warning: Permanently added 'an-a01dr01' (ED25519) to the list of known hosts. [2026-09-04T16:45:41.248Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-09-04T16:45:41.248Z] This might take a few seconds. [2026-09-04T16:45:41.248Z] [2026-09-04T16:45:41.248Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T16:45:42.434Z] ! Cannot disconnect from Red Hat Insights [2026-09-04T16:45:42.434Z] ! Cannot disconnect from Red Hat Subscription Management [2026-09-04T16:45:42.434Z] [2026-09-04T16:45:42.434Z] Manage your connected systems: https://red.ht/connector [2026-09-04T16:45:42.434Z] [2026-09-04T16:45:42.434Z] The following errors were encountered during disconnect: [2026-09-04T16:45:42.434Z] [2026-09-04T16:45:42.434Z] TYPE STEP ERROR [2026-09-04T16:45:42.434Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-09-04T16:45:42.434Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-09-04T16:45:43.619Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T16:45:43.619Z] [2026-09-04T16:45:43.619Z] 1. Register with subscription-manager [2026-09-04T16:45:43.619Z] # subscription-manager register [2026-09-04T16:45:43.619Z] [2026-09-04T16:45:43.619Z] 2. Register with insights-client [2026-09-04T16:45:43.619Z] # insights-client --register [2026-09-04T16:45:44.805Z] This system is currently not registered. [2026-09-04T16:45:47.834Z] Domain 'an-a01dr01' destroyed [2026-09-04T16:45:47.834Z] [2026-09-04T16:45:47.834Z] Domain 'an-a01dr01' has been undefined [2026-09-04T16:45:47.834Z] [2026-09-04T16:45:47.834Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-09-04T16:45:47.834Z] Genering ssh keys and ssh config [2026-09-04T16:45:48.175Z] Cache already updated [2026-09-04T16:45:48.175Z] Creating nodes disks [2026-09-04T16:45:54.192Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-09-04T16:45:54.192Z] Generating an-test-deployX kickstart files/isos [2026-09-04T16:45:54.192Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-09-04T16:45:54.192Z] [2026-09-04T16:45:54.192Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-09-04T16:45:54.192Z] Media current: stdio file, overwriteable [2026-09-04T16:45:54.192Z] Media status : is blank [2026-09-04T16:45:54.192Z] Media summary: 0 sessions, 0 data blocks, 0 data, 801g free [2026-09-04T16:45:54.192Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-09-04T16:45:54.192Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.192Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.192Z] ISO image produced: 184 sectors [2026-09-04T16:45:54.192Z] Written to medium : 184 sectors at LBA 0 [2026-09-04T16:45:54.192Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-09-04T16:45:54.192Z] [2026-09-04T16:45:54.192Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-09-04T16:45:54.192Z] [2026-09-04T16:45:54.192Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-09-04T16:45:54.192Z] Media current: stdio file, overwriteable [2026-09-04T16:45:54.192Z] Media status : is blank [2026-09-04T16:45:54.192Z] Media summary: 0 sessions, 0 data blocks, 0 data, 801g free [2026-09-04T16:45:54.192Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-09-04T16:45:54.192Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.192Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.533Z] ISO image produced: 184 sectors [2026-09-04T16:45:54.533Z] Written to medium : 184 sectors at LBA 0 [2026-09-04T16:45:54.533Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-09-04T16:45:54.533Z] Media current: stdio file, overwriteable [2026-09-04T16:45:54.533Z] Media status : is blank [2026-09-04T16:45:54.533Z] Media summary: 0 sessions, 0 data blocks, 0 data, 801g free [2026-09-04T16:45:54.533Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-09-04T16:45:54.533Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.533Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.533Z] ISO image produced: 184 sectors [2026-09-04T16:45:54.533Z] Written to medium : 184 sectors at LBA 0 [2026-09-04T16:45:54.533Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-09-04T16:45:54.533Z] Media current: stdio file, overwriteable [2026-09-04T16:45:54.533Z] Media status : is blank [2026-09-04T16:45:54.533Z] Media summary: 0 sessions, 0 data blocks, 0 data, 801g free [2026-09-04T16:45:54.533Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-09-04T16:45:54.533Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.533Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.533Z] ISO image produced: 184 sectors [2026-09-04T16:45:54.533Z] Written to medium : 184 sectors at LBA 0 [2026-09-04T16:45:54.533Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-09-04T16:45:54.533Z] [2026-09-04T16:45:54.533Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-09-04T16:45:54.533Z] Media current: stdio file, overwriteable [2026-09-04T16:45:54.533Z] Media status : is blank [2026-09-04T16:45:54.533Z] Media summary: 0 sessions, 0 data blocks, 0 data, 801g free [2026-09-04T16:45:54.533Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-09-04T16:45:54.534Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.534Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-09-04T16:45:54.534Z] ISO image produced: 184 sectors [2026-09-04T16:45:54.534Z] Written to medium : 184 sectors at LBA 0 [2026-09-04T16:45:54.534Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-09-04T16:45:54.534Z] [2026-09-04T16:45:54.874Z] Generating cloud-init configs [2026-09-04T16:45:54.874Z] Generating cloud-init metadata and seed images [2026-09-04T16:45:54.874Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-09-04T16:45:54.874Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-09-04T16:45:55.218Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-09-04T16:45:55.218Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-09-04T16:45:55.218Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-09-04T16:45:55.218Z] Creating VMs [2026-09-04T16:45:56.929Z] [2026-09-04T16:45:56.929Z] Starting install... [2026-09-04T16:45:56.929Z] [2026-09-04T16:45:56.929Z] Creating domain... | 00:00 [2026-09-04T16:45:56.929Z] Domain creation completed. [2026-09-04T16:45:58.114Z] [2026-09-04T16:45:58.114Z] Starting install... [2026-09-04T16:45:58.114Z] [2026-09-04T16:45:58.114Z] Creating domain... | 00:00 [2026-09-04T16:45:58.114Z] Domain creation completed. [2026-09-04T16:45:59.825Z] [2026-09-04T16:45:59.825Z] Starting install... [2026-09-04T16:45:59.825Z] [2026-09-04T16:45:59.825Z] Creating domain... | 00:00 [2026-09-04T16:45:59.825Z] Domain creation completed. [2026-09-04T16:46:01.011Z] [2026-09-04T16:46:01.011Z] Starting install... [2026-09-04T16:46:01.011Z] [2026-09-04T16:46:01.011Z] Creating domain... | 00:00 [2026-09-04T16:46:01.011Z] Domain creation completed. [2026-09-04T16:46:02.728Z] [2026-09-04T16:46:02.728Z] Starting install... [2026-09-04T16:46:02.728Z] [2026-09-04T16:46:02.728Z] Creating domain... | 00:00 [2026-09-04T16:46:02.728Z] Domain creation completed. [2026-09-04T16:46:02.728Z] Waiting for nodes and cloud-init to complete full init [2026-09-04T16:46:05.768Z] cloud-init status node an-striker01: [2026-09-04T16:46:08.814Z] cloud-init status node an-striker02: [2026-09-04T16:46:11.860Z] cloud-init status node an-a01n01: [2026-09-04T16:46:15.147Z] cloud-init status node an-a01n02: [2026-09-04T16:46:18.198Z] cloud-init status node an-a01dr01: [2026-09-04T16:46:18.198Z] Fri Sep 4 12:46:17 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:46:29.599Z] cloud-init status node an-striker01: running [2026-09-04T16:46:30.404Z] cloud-init status node an-striker02: running [2026-09-04T16:46:31.772Z] cloud-init status node an-a01n01: running [2026-09-04T16:46:32.732Z] cloud-init status node an-a01n02: running [2026-09-04T16:46:34.392Z] cloud-init status node an-a01dr01: running [2026-09-04T16:46:34.392Z] Fri Sep 4 12:46:33 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:46:51.353Z] cloud-init status node an-striker01: running [2026-09-04T16:46:51.353Z] cloud-init status node an-striker02: running [2026-09-04T16:46:51.354Z] cloud-init status node an-a01n01: running [2026-09-04T16:46:52.245Z] cloud-init status node an-a01n02: running [2026-09-04T16:46:53.160Z] cloud-init status node an-a01dr01: running [2026-09-04T16:46:53.160Z] Fri Sep 4 12:46:53 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:47:04.887Z] cloud-init status node an-striker01: running [2026-09-04T16:47:04.887Z] cloud-init status node an-striker02: running [2026-09-04T16:47:05.809Z] cloud-init status node an-a01n01: running [2026-09-04T16:47:06.865Z] cloud-init status node an-a01n02: running [2026-09-04T16:47:08.094Z] cloud-init status node an-a01dr01: running [2026-09-04T16:47:08.094Z] Fri Sep 4 12:47:07 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:47:20.339Z] cloud-init status node an-striker01: running [2026-09-04T16:47:20.339Z] cloud-init status node an-striker02: running [2026-09-04T16:47:20.339Z] cloud-init status node an-a01n01: running [2026-09-04T16:47:21.625Z] cloud-init status node an-a01n02: running [2026-09-04T16:47:22.493Z] cloud-init status node an-a01dr01: running [2026-09-04T16:47:22.493Z] Fri Sep 4 12:47:22 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:47:33.193Z] cloud-init status node an-striker01: running [2026-09-04T16:47:34.007Z] cloud-init status node an-striker02: running [2026-09-04T16:47:35.260Z] cloud-init status node an-a01n01: running [2026-09-04T16:47:37.286Z] cloud-init status node an-a01n02: running [2026-09-04T16:47:38.085Z] cloud-init status node an-a01dr01: running [2026-09-04T16:47:38.085Z] Fri Sep 4 12:47:37 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:47:51.751Z] cloud-init status node an-striker01: running [2026-09-04T16:47:51.751Z] cloud-init status node an-striker02: running [2026-09-04T16:47:51.751Z] cloud-init status node an-a01n01: running [2026-09-04T16:47:51.751Z] cloud-init status node an-a01n02: running [2026-09-04T16:47:52.772Z] cloud-init status node an-a01dr01: running [2026-09-04T16:47:52.772Z] Fri Sep 4 12:47:52 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:48:04.981Z] cloud-init status node an-striker01: running [2026-09-04T16:48:04.981Z] cloud-init status node an-striker02: running [2026-09-04T16:48:10.519Z] cloud-init status node an-a01n01: running [2026-09-04T16:48:11.596Z] cloud-init status node an-a01n02: running [2026-09-04T16:48:12.587Z] cloud-init status node an-a01dr01: running [2026-09-04T16:48:12.587Z] Fri Sep 4 12:48:12 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:48:23.663Z] cloud-init status node an-striker01: running [2026-09-04T16:48:24.417Z] cloud-init status node an-striker02: running [2026-09-04T16:48:25.467Z] cloud-init status node an-a01n01: running [2026-09-04T16:48:26.252Z] cloud-init status node an-a01n02: running [2026-09-04T16:48:26.894Z] cloud-init status node an-a01dr01: running [2026-09-04T16:48:26.894Z] Fri Sep 4 12:48:26 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:48:44.610Z] cloud-init status node an-striker01: running [2026-09-04T16:48:44.610Z] cloud-init status node an-striker02: running [2026-09-04T16:48:44.986Z] cloud-init status node an-a01n01: running [2026-09-04T16:48:46.121Z] cloud-init status node an-a01n02: running [2026-09-04T16:48:46.944Z] cloud-init status node an-a01dr01: running [2026-09-04T16:48:46.944Z] Fri Sep 4 12:48:46 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:48:58.826Z] cloud-init status node an-striker01: running [2026-09-04T16:48:58.826Z] cloud-init status node an-striker02: running [2026-09-04T16:48:59.713Z] cloud-init status node an-a01n01: running [2026-09-04T16:49:00.701Z] cloud-init status node an-a01n02: running [2026-09-04T16:49:01.818Z] cloud-init status node an-a01dr01: running [2026-09-04T16:49:01.818Z] Fri Sep 4 12:49:01 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:49:14.163Z] cloud-init status node an-striker01: running [2026-09-04T16:49:14.163Z] cloud-init status node an-striker02: running [2026-09-04T16:49:14.780Z] cloud-init status node an-a01n01: running [2026-09-04T16:49:15.282Z] cloud-init status node an-a01n02: running [2026-09-04T16:49:21.669Z] cloud-init status node an-a01dr01: running [2026-09-04T16:49:21.669Z] Fri Sep 4 12:49:21 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:49:32.362Z] cloud-init status node an-striker01: running [2026-09-04T16:49:33.164Z] cloud-init status node an-striker02: running [2026-09-04T16:49:34.450Z] cloud-init status node an-a01n01: running [2026-09-04T16:49:35.356Z] cloud-init status node an-a01n02: running [2026-09-04T16:49:35.959Z] cloud-init status node an-a01dr01: running [2026-09-04T16:49:35.959Z] Fri Sep 4 12:49:35 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:49:49.105Z] cloud-init status node an-striker01: running [2026-09-04T16:49:54.211Z] cloud-init status node an-striker02: running [2026-09-04T16:49:54.211Z] cloud-init status node an-a01n01: running [2026-09-04T16:49:55.074Z] cloud-init status node an-a01n02: running [2026-09-04T16:49:56.541Z] cloud-init status node an-a01dr01: running [2026-09-04T16:49:56.541Z] Fri Sep 4 12:49:55 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:50:07.465Z] cloud-init status node an-striker01: running [2026-09-04T16:50:08.201Z] cloud-init status node an-striker02: running [2026-09-04T16:50:09.051Z] cloud-init status node an-a01n01: running [2026-09-04T16:50:10.257Z] cloud-init status node an-a01n02: running [2026-09-04T16:50:10.815Z] cloud-init status node an-a01dr01: running [2026-09-04T16:50:10.815Z] Fri Sep 4 12:50:10 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:50:21.856Z] cloud-init status node an-striker01: running [2026-09-04T16:50:23.309Z] cloud-init status node an-striker02: running [2026-09-04T16:50:24.218Z] cloud-init status node an-a01n01: running [2026-09-04T16:50:25.325Z] cloud-init status node an-a01n02: running [2026-09-04T16:50:25.715Z] cloud-init status node an-a01dr01: running [2026-09-04T16:50:25.715Z] Fri Sep 4 12:50:25 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:50:38.202Z] cloud-init status node an-striker01: running [2026-09-04T16:50:38.202Z] cloud-init status node an-striker02: running [2026-09-04T16:50:39.138Z] cloud-init status node an-a01n01: running [2026-09-04T16:50:39.990Z] cloud-init status node an-a01n02: running [2026-09-04T16:50:40.739Z] cloud-init status node an-a01dr01: running [2026-09-04T16:50:40.739Z] Fri Sep 4 12:50:40 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:50:52.897Z] cloud-init status node an-striker01: running [2026-09-04T16:50:57.648Z] cloud-init status node an-striker02: running [2026-09-04T16:50:58.544Z] cloud-init status node an-a01n01: running [2026-09-04T16:51:00.114Z] cloud-init status node an-a01n02: running [2026-09-04T16:51:00.510Z] cloud-init status node an-a01dr01: running [2026-09-04T16:51:00.510Z] Fri Sep 4 12:51:00 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:51:12.415Z] cloud-init status node an-striker01: running [2026-09-04T16:51:12.415Z] cloud-init status node an-striker02: running [2026-09-04T16:51:13.612Z] cloud-init status node an-a01n01: running [2026-09-04T16:51:14.450Z] cloud-init status node an-a01n02: running [2026-09-04T16:51:15.522Z] cloud-init status node an-a01dr01: running [2026-09-04T16:51:15.522Z] Fri Sep 4 12:51:15 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:51:27.568Z] cloud-init status node an-striker01: running [2026-09-04T16:51:27.568Z] cloud-init status node an-striker02: running [2026-09-04T16:51:28.352Z] cloud-init status node an-a01n01: running [2026-09-04T16:51:29.116Z] cloud-init status node an-a01n02: running [2026-09-04T16:51:29.883Z] cloud-init status node an-a01dr01: running [2026-09-04T16:51:29.883Z] Fri Sep 4 12:51:29 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:51:41.031Z] cloud-init status node an-striker01: running [2026-09-04T16:51:41.770Z] cloud-init status node an-striker02: running [2026-09-04T16:51:42.989Z] cloud-init status node an-a01n01: running [2026-09-04T16:51:43.728Z] cloud-init status node an-a01n02: running [2026-09-04T16:51:44.920Z] cloud-init status node an-a01dr01: running [2026-09-04T16:51:44.920Z] Fri Sep 4 12:51:44 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:51:55.920Z] cloud-init status node an-striker01: running [2026-09-04T16:51:56.654Z] cloud-init status node an-striker02: running [2026-09-04T16:51:57.838Z] cloud-init status node an-a01n01: running [2026-09-04T16:51:58.571Z] cloud-init status node an-a01n02: running [2026-09-04T16:51:59.756Z] cloud-init status node an-a01dr01: running [2026-09-04T16:51:59.756Z] Fri Sep 4 12:51:59 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:52:10.732Z] cloud-init status node an-striker01: running [2026-09-04T16:52:11.465Z] cloud-init status node an-striker02: running [2026-09-04T16:52:12.198Z] cloud-init status node an-a01n01: running [2026-09-04T16:52:13.604Z] cloud-init status node an-a01n02: running [2026-09-04T16:52:14.336Z] cloud-init status node an-a01dr01: running [2026-09-04T16:52:14.336Z] Fri Sep 4 12:52:14 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:52:27.621Z] cloud-init status node an-striker01: running [2026-09-04T16:52:27.621Z] cloud-init status node an-striker02: running [2026-09-04T16:52:27.963Z] cloud-init status node an-a01n01: running [2026-09-04T16:52:29.148Z] cloud-init status node an-a01n02: running [2026-09-04T16:52:30.333Z] cloud-init status node an-a01dr01: running [2026-09-04T16:52:30.333Z] Fri Sep 4 12:52:30 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:52:41.304Z] cloud-init status node an-striker01: running [2026-09-04T16:52:42.036Z] cloud-init status node an-striker02: running [2026-09-04T16:52:42.770Z] cloud-init status node an-a01n01: running [2026-09-04T16:52:43.954Z] cloud-init status node an-a01n02: running [2026-09-04T16:52:44.688Z] cloud-init status node an-a01dr01: running [2026-09-04T16:52:44.688Z] Fri Sep 4 12:52:44 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:52:55.653Z] cloud-init status node an-striker01: running [2026-09-04T16:52:56.387Z] cloud-init status node an-striker02: running [2026-09-04T16:52:57.571Z] cloud-init status node an-a01n01: running [2026-09-04T16:52:58.303Z] cloud-init status node an-a01n02: running [2026-09-04T16:52:59.487Z] cloud-init status node an-a01dr01: running [2026-09-04T16:52:59.487Z] Fri Sep 4 12:52:59 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:53:10.475Z] cloud-init status node an-striker01: running [2026-09-04T16:53:11.209Z] cloud-init status node an-striker02: running [2026-09-04T16:53:12.394Z] cloud-init status node an-a01n01: running [2026-09-04T16:53:13.129Z] cloud-init status node an-a01n02: running [2026-09-04T16:53:14.314Z] cloud-init status node an-a01dr01: running [2026-09-04T16:53:14.314Z] Fri Sep 4 12:53:14 EDT 2026 actual: 0 expected: 5 [2026-09-04T16:53:27.603Z] cloud-init status node an-striker01: running [2026-09-04T16:53:27.603Z] cloud-init status node an-striker02: running [2026-09-04T16:53:27.603Z] cloud-init status node an-a01n01: done [2026-09-04T16:53:28.789Z] cloud-init status node an-a01n02: running [2026-09-04T16:53:29.705Z] cloud-init status node an-a01dr01: done [2026-09-04T16:53:29.705Z] Fri Sep 4 12:53:29 EDT 2026 actual: 2 expected: 5 [2026-09-04T16:53:40.683Z] cloud-init status node an-striker01: running [2026-09-04T16:53:41.870Z] cloud-init status node an-striker02: running [2026-09-04T16:53:42.603Z] cloud-init status node an-a01n01: done [2026-09-04T16:53:43.787Z] cloud-init status node an-a01n02: running [2026-09-04T16:53:44.972Z] cloud-init status node an-a01dr01: done [2026-09-04T16:53:44.972Z] Fri Sep 4 12:53:44 EDT 2026 actual: 2 expected: 5 [2026-09-04T16:53:55.944Z] cloud-init status node an-striker01: running [2026-09-04T16:53:56.678Z] cloud-init status node an-striker02: running [2026-09-04T16:53:57.865Z] cloud-init status node an-a01n01: done [2026-09-04T16:53:58.207Z] cloud-init status node an-a01n02: done [2026-09-04T16:53:59.393Z] cloud-init status node an-a01dr01: done [2026-09-04T16:53:59.393Z] Fri Sep 4 12:53:59 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:54:10.365Z] cloud-init status node an-striker01: running [2026-09-04T16:54:11.549Z] cloud-init status node an-striker02: running [2026-09-04T16:54:12.282Z] cloud-init status node an-a01n01: done [2026-09-04T16:54:13.466Z] cloud-init status node an-a01n02: done [2026-09-04T16:54:14.200Z] cloud-init status node an-a01dr01: done [2026-09-04T16:54:14.200Z] Fri Sep 4 12:54:13 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:54:25.170Z] cloud-init status node an-striker01: running [2026-09-04T16:54:25.903Z] cloud-init status node an-striker02: running [2026-09-04T16:54:27.088Z] cloud-init status node an-a01n01: done [2026-09-04T16:54:27.821Z] cloud-init status node an-a01n02: done [2026-09-04T16:54:29.004Z] cloud-init status node an-a01dr01: done [2026-09-04T16:54:29.004Z] Fri Sep 4 12:54:28 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:54:39.973Z] cloud-init status node an-striker01: running [2026-09-04T16:54:41.159Z] cloud-init status node an-striker02: running [2026-09-04T16:54:41.892Z] cloud-init status node an-a01n01: done [2026-09-04T16:54:42.675Z] cloud-init status node an-a01n02: done [2026-09-04T16:54:43.861Z] cloud-init status node an-a01dr01: done [2026-09-04T16:54:43.861Z] Fri Sep 4 12:54:43 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:54:57.145Z] cloud-init status node an-striker01: running [2026-09-04T16:54:57.145Z] cloud-init status node an-striker02: running [2026-09-04T16:54:57.879Z] cloud-init status node an-a01n01: done [2026-09-04T16:54:58.612Z] cloud-init status node an-a01n02: done [2026-09-04T16:54:59.798Z] cloud-init status node an-a01dr01: done [2026-09-04T16:54:59.798Z] Fri Sep 4 12:54:59 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:55:10.774Z] cloud-init status node an-striker01: running [2026-09-04T16:55:11.962Z] cloud-init status node an-striker02: running [2026-09-04T16:55:12.697Z] cloud-init status node an-a01n01: done [2026-09-04T16:55:13.891Z] cloud-init status node an-a01n02: done [2026-09-04T16:55:14.624Z] cloud-init status node an-a01dr01: done [2026-09-04T16:55:14.624Z] Fri Sep 4 12:55:14 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:55:27.912Z] cloud-init status node an-striker01: running [2026-09-04T16:55:27.912Z] cloud-init status node an-striker02: running [2026-09-04T16:55:28.646Z] cloud-init status node an-a01n01: done [2026-09-04T16:55:29.381Z] cloud-init status node an-a01n02: done [2026-09-04T16:55:30.567Z] cloud-init status node an-a01dr01: done [2026-09-04T16:55:30.567Z] Fri Sep 4 12:55:30 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:55:41.539Z] cloud-init status node an-striker01: running [2026-09-04T16:55:42.722Z] cloud-init status node an-striker02: running [2026-09-04T16:55:43.907Z] cloud-init status node an-a01n01: done [2026-09-04T16:55:44.640Z] cloud-init status node an-a01n02: done [2026-09-04T16:55:45.824Z] cloud-init status node an-a01dr01: done [2026-09-04T16:55:45.824Z] Fri Sep 4 12:55:45 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:55:56.791Z] cloud-init status node an-striker01: running [2026-09-04T16:55:57.976Z] cloud-init status node an-striker02: running [2026-09-04T16:55:59.205Z] cloud-init status node an-a01n01: done [2026-09-04T16:55:59.937Z] cloud-init status node an-a01n02: done [2026-09-04T16:56:01.122Z] cloud-init status node an-a01dr01: done [2026-09-04T16:56:01.122Z] Fri Sep 4 12:56:00 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:56:12.089Z] cloud-init status node an-striker01: running [2026-09-04T16:56:13.271Z] cloud-init status node an-striker02: running [2026-09-04T16:56:14.004Z] cloud-init status node an-a01n01: done [2026-09-04T16:56:14.736Z] cloud-init status node an-a01n02: done [2026-09-04T16:56:15.919Z] cloud-init status node an-a01dr01: done [2026-09-04T16:56:15.919Z] Fri Sep 4 12:56:15 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:56:26.888Z] cloud-init status node an-striker01: running [2026-09-04T16:56:28.072Z] cloud-init status node an-striker02: running [2026-09-04T16:56:28.805Z] cloud-init status node an-a01n01: done [2026-09-04T16:56:29.988Z] cloud-init status node an-a01n02: done [2026-09-04T16:56:30.720Z] cloud-init status node an-a01dr01: done [2026-09-04T16:56:30.720Z] Fri Sep 4 12:56:30 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:56:44.008Z] cloud-init status node an-striker01: running [2026-09-04T16:56:44.008Z] cloud-init status node an-striker02: running [2026-09-04T16:56:44.743Z] cloud-init status node an-a01n01: done [2026-09-04T16:56:45.927Z] cloud-init status node an-a01n02: done [2026-09-04T16:56:46.660Z] cloud-init status node an-a01dr01: done [2026-09-04T16:56:46.660Z] Fri Sep 4 12:56:46 EDT 2026 actual: 3 expected: 5 [2026-09-04T16:56:57.630Z] cloud-init status node an-striker01: done [2026-09-04T16:56:59.339Z] cloud-init status node an-striker02: done [2026-09-04T16:57:00.072Z] cloud-init status node an-a01n01: done [2026-09-04T16:57:00.806Z] cloud-init status node an-a01n02: done [2026-09-04T16:57:01.991Z] cloud-init status node an-a01dr01: done [2026-09-04T16:57:01.991Z] Fri Sep 4 12:57:01 EDT 2026 actual: 5 expected: 5 [2026-09-04T16:57:01.991Z] Building simengine model [2026-09-04T16:57:15.414Z] Generating anvil striker auto config [2026-09-04T16:57:17.753Z] Cloning into 'anvil-test-suite'... [2026-09-04T16:57:18.094Z] generated ansible config: [2026-09-04T16:57:18.094Z] ========================= [2026-09-04T16:57:18.094Z] [defaults] [2026-09-04T16:57:18.094Z] inventory = ansible-inventory [2026-09-04T16:57:18.094Z] forks = 500 [2026-09-04T16:57:18.094Z] any_errors_fatal = true [2026-09-04T16:57:18.094Z] display_args_to_stdout = true [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:18.094Z] [ssh_connection] [2026-09-04T16:57:18.094Z] retries = 20 [2026-09-04T16:57:18.094Z] ========================= [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:18.094Z] generated ansible inventory: [2026-09-04T16:57:18.094Z] ============================ [2026-09-04T16:57:18.094Z] [strikers] [2026-09-04T16:57:18.094Z] an-striker01 [2026-09-04T16:57:18.094Z] an-striker02 [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:18.094Z] [nodes] [2026-09-04T16:57:18.094Z] an-a01n01 [2026-09-04T16:57:18.094Z] an-a01n02 [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:18.094Z] [drnodes] [2026-09-04T16:57:18.094Z] an-a01dr01 [2026-09-04T16:57:18.094Z] ============================ [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:18.094Z] Checking connectivity to nodes [2026-09-04T16:57:18.094Z] [2026-09-04T16:57:20.412Z] an-a01n02 | SUCCESS => { [2026-09-04T16:57:20.412Z] "ansible_facts": { [2026-09-04T16:57:20.412Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-09-04T16:57:20.412Z] }, [2026-09-04T16:57:20.412Z] "changed": false, [2026-09-04T16:57:20.412Z] "ping": "pong" [2026-09-04T16:57:20.412Z] } [2026-09-04T16:57:20.412Z] an-a01n01 | SUCCESS => { [2026-09-04T16:57:20.412Z] "ansible_facts": { [2026-09-04T16:57:20.412Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-09-04T16:57:20.412Z] }, [2026-09-04T16:57:20.412Z] "changed": false, [2026-09-04T16:57:20.412Z] "ping": "pong" [2026-09-04T16:57:20.412Z] } [2026-09-04T16:57:20.412Z] an-a01dr01 | SUCCESS => { [2026-09-04T16:57:20.412Z] "ansible_facts": { [2026-09-04T16:57:20.412Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-09-04T16:57:20.412Z] }, [2026-09-04T16:57:20.412Z] "changed": false, [2026-09-04T16:57:20.412Z] "ping": "pong" [2026-09-04T16:57:20.412Z] } [2026-09-04T16:57:20.412Z] an-striker01 | SUCCESS => { [2026-09-04T16:57:20.412Z] "ansible_facts": { [2026-09-04T16:57:20.412Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-09-04T16:57:20.412Z] }, [2026-09-04T16:57:20.412Z] "changed": false, [2026-09-04T16:57:20.412Z] "ping": "pong" [2026-09-04T16:57:20.412Z] } [2026-09-04T16:57:20.412Z] an-striker02 | SUCCESS => { [2026-09-04T16:57:20.412Z] "ansible_facts": { [2026-09-04T16:57:20.412Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-09-04T16:57:20.412Z] }, [2026-09-04T16:57:20.412Z] "changed": false, [2026-09-04T16:57:20.412Z] "ping": "pong" [2026-09-04T16:57:20.412Z] } [2026-09-04T16:57:20.412Z] [2026-09-04T16:57:21.146Z] [2026-09-04T16:57:21.146Z] PLAY [Setup all nodes] ********************************************************* [2026-09-04T16:57:21.146Z] [2026-09-04T16:57:21.146Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T16:57:23.466Z] ok: [an-a01n01] [2026-09-04T16:57:23.466Z] ok: [an-a01n02] [2026-09-04T16:57:23.466Z] ok: [an-a01dr01] [2026-09-04T16:57:23.466Z] ok: [an-striker01] [2026-09-04T16:57:23.466Z] ok: [an-striker02] [2026-09-04T16:57:23.466Z] [2026-09-04T16:57:23.466Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-09-04T16:57:24.656Z] changed: [an-a01n01] [2026-09-04T16:57:24.656Z] changed: [an-a01n02] [2026-09-04T16:57:24.656Z] changed: [an-a01dr01] [2026-09-04T16:57:24.656Z] changed: [an-striker02] [2026-09-04T16:57:24.656Z] changed: [an-striker01] [2026-09-04T16:57:24.656Z] [2026-09-04T16:57:24.656Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-09-04T16:57:25.391Z] ok: [an-a01n01] [2026-09-04T16:57:25.391Z] ok: [an-a01n02] [2026-09-04T16:57:25.391Z] ok: [an-a01dr01] [2026-09-04T16:57:25.391Z] ok: [an-striker01] [2026-09-04T16:57:25.391Z] ok: [an-striker02] [2026-09-04T16:57:25.391Z] [2026-09-04T16:57:25.391Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-09-04T16:57:26.575Z] changed: [an-a01n01] [2026-09-04T16:57:26.575Z] changed: [an-a01dr01] [2026-09-04T16:57:26.575Z] changed: [an-a01n02] [2026-09-04T16:57:26.575Z] changed: [an-striker01] [2026-09-04T16:57:26.575Z] changed: [an-striker02] [2026-09-04T16:57:26.575Z] [2026-09-04T16:57:26.575Z] PLAY [setup strikers] ********************************************************** [2026-09-04T16:57:26.575Z] [2026-09-04T16:57:26.575Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T16:57:28.284Z] ok: [an-striker01] [2026-09-04T16:57:28.284Z] ok: [an-striker02] [2026-09-04T16:57:28.284Z] [2026-09-04T16:57:28.284Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-09-04T16:57:29.992Z] changed: [an-striker02] [2026-09-04T16:57:29.992Z] changed: [an-striker01] [2026-09-04T16:57:29.992Z] [2026-09-04T16:57:29.992Z] PLAY [setup lvm storage] ******************************************************* [2026-09-04T16:57:29.992Z] [2026-09-04T16:57:29.992Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T16:57:31.700Z] ok: [an-a01n01] [2026-09-04T16:57:31.700Z] ok: [an-a01n02] [2026-09-04T16:57:31.700Z] ok: [an-a01dr01] [2026-09-04T16:57:31.700Z] [2026-09-04T16:57:31.700Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-09-04T16:57:31.700Z] ] **************** [2026-09-04T16:57:32.434Z] ok: [an-a01n01] [2026-09-04T16:57:32.434Z] ok: [an-a01n02] [2026-09-04T16:57:32.434Z] ok: [an-a01dr01] [2026-09-04T16:57:32.434Z] [2026-09-04T16:57:32.434Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-09-04T16:57:32.434Z] skipping: [an-a01n01] [2026-09-04T16:57:32.434Z] [2026-09-04T16:57:32.434Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-09-04T16:57:33.618Z] changed: [an-a01n01] [2026-09-04T16:57:33.618Z] changed: [an-a01n02] [2026-09-04T16:57:33.618Z] changed: [an-a01dr01] [2026-09-04T16:57:33.618Z] [2026-09-04T16:57:33.618Z] PLAY [deploy anvil] ************************************************************ [2026-09-04T16:57:33.618Z] [2026-09-04T16:57:33.618Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T16:57:34.802Z] ok: [an-striker01] [2026-09-04T16:57:34.802Z] ok: [an-striker02] [2026-09-04T16:57:34.802Z] [2026-09-04T16:57:34.802Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-09-04T16:57:35.535Z] ok: [an-striker01] [2026-09-04T16:57:35.535Z] ok: [an-striker02] [2026-09-04T16:57:35.535Z] [2026-09-04T16:57:35.535Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-09-04T16:57:37.245Z] changed: [an-striker01] [2026-09-04T16:57:37.245Z] changed: [an-striker02] [2026-09-04T16:57:37.245Z] [2026-09-04T16:57:37.245Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-09-04T16:57:37.245Z] ] *** [2026-09-04T16:57:40.278Z] changed: [an-striker02] [2026-09-04T16:57:40.278Z] changed: [an-striker01] [2026-09-04T16:57:40.278Z] [2026-09-04T16:57:40.278Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T16:57:40.278Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T16:57:40.278Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-09-04T16:57:40.278Z] ] *** [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-09-04T17:03:35.309Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-09-04T17:03:35.309Z] ok: [an-striker01 -> localhost] [2026-09-04T17:03:35.309Z] ok: [an-striker02 -> localhost] [2026-09-04T17:03:35.309Z] [2026-09-04T17:03:35.309Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-09-04T17:03:35.309Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-09-04T17:03:35.309Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-09-04T17:03:35.309Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-09-04T17:03:35.309Z] echo "DB has completed sync" [2026-09-04T17:03:35.309Z] exit 0 [2026-09-04T17:03:35.309Z] fi [2026-09-04T17:03:35.309Z] if [ "$dbcon" != 2 ]; then [2026-09-04T17:03:35.309Z] echo "Detected $dbcon db connections (expected 2)" [2026-09-04T17:03:35.309Z] exit 1 [2026-09-04T17:03:35.309Z] fi [2026-09-04T17:03:35.309Z] ] *** [2026-09-04T17:04:23.759Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-09-04T17:04:23.759Z] ok: [an-striker01] [2026-09-04T17:04:23.759Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-09-04T17:04:23.759Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left). [2026-09-04T17:04:23.759Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (117 retries left). [2026-09-04T17:04:23.759Z] ok: [an-striker02] [2026-09-04T17:04:23.759Z] [2026-09-04T17:04:23.759Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T17:04:23.759Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T17:04:23.759Z] --username=postgres anvil -t \ [2026-09-04T17:04:23.759Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-09-04T17:04:23.759Z] grep altee | \ [2026-09-04T17:04:23.759Z] grep 1$ | \ [2026-09-04T17:04:23.759Z] wc -l [2026-09-04T17:04:23.759Z] ] *** [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-09-04T17:10:03.699Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-09-04T17:13:57.992Z] ok: [an-striker01 -> localhost] [2026-09-04T17:13:57.992Z] ok: [an-striker02 -> localhost] [2026-09-04T17:13:57.992Z] [2026-09-04T17:13:57.992Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T17:13:57.992Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T17:13:57.992Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-09-04T17:13:57.992Z] grep altee | \ [2026-09-04T17:13:57.992Z] wc -l [2026-09-04T17:13:57.992Z] ] *** [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-09-04T17:13:57.992Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-09-04T17:13:57.992Z] ok: [an-striker01 -> localhost] [2026-09-04T17:13:57.992Z] ok: [an-striker02 -> localhost] [2026-09-04T17:13:57.992Z] [2026-09-04T17:13:57.992Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T17:13:57.992Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T17:13:57.992Z] --username=postgres anvil -t \ [2026-09-04T17:13:57.992Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-09-04T17:13:57.992Z] grep altee | \ [2026-09-04T17:13:57.992Z] grep 1$ | \ [2026-09-04T17:13:57.992Z] wc -l [2026-09-04T17:13:57.992Z] ] *** [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-09-04T17:15:51.710Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-09-04T17:15:51.710Z] ok: [an-striker01 -> localhost] [2026-09-04T17:15:51.710Z] ok: [an-striker02 -> localhost] [2026-09-04T17:15:51.710Z] [2026-09-04T17:15:51.710Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T17:15:51.710Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T17:15:51.710Z] --username=postgres anvil -t \ [2026-09-04T17:15:51.710Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-09-04T17:15:51.710Z] grep altee | \ [2026-09-04T17:15:51.710Z] grep 1$ | \ [2026-09-04T17:15:51.710Z] wc -l [2026-09-04T17:15:51.710Z] ] *** [2026-09-04T17:16:02.683Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-09-04T17:16:02.683Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-09-04T17:16:02.683Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-09-04T17:16:02.683Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-09-04T17:16:02.683Z] ok: [an-striker01 -> localhost] [2026-09-04T17:16:02.683Z] ok: [an-striker02 -> localhost] [2026-09-04T17:16:02.683Z] [2026-09-04T17:16:02.683Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-09-04T17:16:02.683Z] psql --host="{{ inventory_hostname }}" \ [2026-09-04T17:16:02.683Z] --username=postgres anvil -t \ [2026-09-04T17:16:02.683Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-09-04T17:16:02.683Z] grep altee | \ [2026-09-04T17:16:02.683Z] grep 1$ | \ [2026-09-04T17:16:02.683Z] wc -l [2026-09-04T17:16:02.683Z] ] *** [2026-09-04T17:16:02.683Z] ok: [an-striker01 -> localhost] [2026-09-04T17:16:02.683Z] ok: [an-striker02 -> localhost] [2026-09-04T17:16:02.683Z] [2026-09-04T17:16:02.683Z] PLAY [check network status after deployment] *********************************** [2026-09-04T17:16:02.683Z] [2026-09-04T17:16:02.683Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:16:04.394Z] ok: [an-a01n02] [2026-09-04T17:16:04.394Z] ok: [an-a01n01] [2026-09-04T17:16:04.394Z] ok: [an-striker02] [2026-09-04T17:16:04.394Z] ok: [an-a01dr01] [2026-09-04T17:16:04.394Z] ok: [an-striker01] [2026-09-04T17:16:04.394Z] [2026-09-04T17:16:04.394Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-09-04T17:16:04.394Z] if [ "$device" = "" ]; then [2026-09-04T17:16:04.394Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-09-04T17:16:04.394Z] exit 1 [2026-09-04T17:16:04.394Z] fi [2026-09-04T17:16:04.394Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-09-04T17:16:04.394Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-09-04T17:16:04.394Z] exit 1 [2026-09-04T17:16:04.394Z] fi [2026-09-04T17:16:04.394Z] exit 0 [2026-09-04T17:16:04.394Z] ] *** [2026-09-04T17:16:04.394Z] skipping: [an-striker01] => (item=lo) [2026-09-04T17:16:04.394Z] skipping: [an-a01n01] => (item=lo) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=sn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=sn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=sn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=ifn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=bcn1_link1) [2026-09-04T17:16:13.431Z] skipping: [an-striker02] => (item=lo) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=ifn1_link2) [2026-09-04T17:16:13.431Z] skipping: [an-a01dr01] => (item=lo) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=bcn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=sn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=ifn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=sn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=bcn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=ifn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=sn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=bcn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=sn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=bcn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=ifn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=ifn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-striker01] => (item=bcn1_link1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=bcn1_bridge1) [2026-09-04T17:16:13.431Z] changed: [an-striker02] => (item=ifn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-09-04T17:16:13.431Z] skipping: [an-a01n02] => (item=lo) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=ifn1_bridge1) [2026-09-04T17:16:13.431Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-09-04T17:16:13.431Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-09-04T17:16:13.431Z] changed: [an-a01dr01] => (item=bcn1_bond1) [2026-09-04T17:16:13.431Z] [2026-09-04T17:16:13.431Z] PLAY [check pacemaker status after deployment] ********************************* [2026-09-04T17:16:13.431Z] [2026-09-04T17:16:13.431Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:16:14.165Z] ok: [an-a01n01] [2026-09-04T17:16:14.165Z] ok: [an-a01n02] [2026-09-04T17:16:14.165Z] [2026-09-04T17:16:14.165Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-09-04T17:16:14.165Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T17:16:14.165Z] [2026-09-04T17:16:14.165Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T17:16:14.165Z] ] *** [2026-09-04T17:16:18.035Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T17:16:18.035Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T17:16:18.035Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T17:16:18.035Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T17:16:18.035Z] [2026-09-04T17:16:18.035Z] PLAY [check drbd kmod] ********************************************************* [2026-09-04T17:16:18.035Z] [2026-09-04T17:16:18.035Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:16:20.352Z] ok: [an-a01n01] [2026-09-04T17:16:20.352Z] ok: [an-a01n02] [2026-09-04T17:16:20.352Z] ok: [an-a01dr01] [2026-09-04T17:16:20.352Z] [2026-09-04T17:16:20.352Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-09-04T17:16:20.352Z] ] *** [2026-09-04T17:16:20.694Z] ok: [an-a01n01] [2026-09-04T17:16:20.694Z] ok: [an-a01n02] [2026-09-04T17:16:20.694Z] ok: [an-a01dr01] [2026-09-04T17:16:20.694Z] [2026-09-04T17:16:20.694Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-09-04T17:16:20.694Z] ] *********************** [2026-09-04T17:16:20.694Z] skipping: [an-a01n01] [2026-09-04T17:16:20.694Z] skipping: [an-a01n02] [2026-09-04T17:16:20.694Z] skipping: [an-a01dr01] [2026-09-04T17:16:20.694Z] [2026-09-04T17:16:20.694Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-09-04T17:16:20.694Z] ] *** [2026-09-04T17:16:20.694Z] skipping: [an-a01n01] [2026-09-04T17:16:21.037Z] skipping: [an-a01n02] [2026-09-04T17:16:21.037Z] skipping: [an-a01dr01] [2026-09-04T17:16:21.037Z] [2026-09-04T17:16:21.037Z] PLAY [Copy iso images to striker01] ******************************************** [2026-09-04T17:16:21.037Z] [2026-09-04T17:16:21.037Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:16:22.747Z] ok: [an-striker01] [2026-09-04T17:16:22.747Z] [2026-09-04T17:16:22.747Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-09-04T17:16:23.480Z] ok: [an-striker01] [2026-09-04T17:16:23.480Z] [2026-09-04T17:16:23.480Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-09-04T17:16:24.667Z] changed: [an-striker01] [2026-09-04T17:16:24.667Z] [2026-09-04T17:16:24.667Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-09-04T17:16:32.064Z] changed: [an-striker01] => (item=deploy1.iso) [2026-09-04T17:16:32.064Z] changed: [an-striker01] => (item=deploy2.iso) [2026-09-04T17:16:32.064Z] changed: [an-striker01] => (item=deploy3.iso) [2026-09-04T17:16:32.064Z] changed: [an-striker01] => (item=deploy4.iso) [2026-09-04T17:16:32.064Z] changed: [an-striker01] => (item=deploy5.iso) [2026-09-04T17:16:32.064Z] [2026-09-04T17:16:32.064Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-09-04T17:16:32.064Z] ] *** [2026-09-04T17:16:33.248Z] changed: [an-striker01] [2026-09-04T17:16:33.248Z] [2026-09-04T17:16:33.248Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-09-04T17:16:34.432Z] changed: [an-striker01] [2026-09-04T17:16:34.432Z] [2026-09-04T17:16:34.432Z] PLAY [Check iso images propagation] ******************************************** [2026-09-04T17:16:34.432Z] [2026-09-04T17:16:34.432Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:16:36.749Z] ok: [an-a01n02] [2026-09-04T17:16:36.749Z] ok: [an-a01n01] [2026-09-04T17:16:36.749Z] ok: [an-striker01] [2026-09-04T17:16:36.749Z] ok: [an-a01dr01] [2026-09-04T17:16:36.749Z] ok: [an-striker02] [2026-09-04T17:16:36.749Z] [2026-09-04T17:16:36.749Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-09-04T17:16:36.749Z] exit 1 [2026-09-04T17:16:36.749Z] else [2026-09-04T17:16:36.749Z] exit 0 [2026-09-04T17:16:36.749Z] fi [2026-09-04T17:16:36.749Z] ] *** [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-09-04T17:18:30.305Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-09-04T17:18:30.305Z] ok: [an-a01n01] [2026-09-04T17:18:30.305Z] ok: [an-a01n02] [2026-09-04T17:18:30.305Z] ok: [an-a01dr01] [2026-09-04T17:18:30.305Z] ok: [an-striker01] [2026-09-04T17:18:30.305Z] ok: [an-striker02] [2026-09-04T17:18:30.305Z] [2026-09-04T17:18:30.305Z] PLAY [example tests] *********************************************************** [2026-09-04T17:18:30.305Z] [2026-09-04T17:18:30.305Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:18:31.610Z] ok: [an-a01n01] [2026-09-04T17:18:31.610Z] ok: [an-a01n02] [2026-09-04T17:18:31.610Z] ok: [an-striker01] [2026-09-04T17:18:31.610Z] ok: [an-a01dr01] [2026-09-04T17:18:31.610Z] ok: [an-striker02] [2026-09-04T17:18:31.610Z] [2026-09-04T17:18:31.610Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-09-04T17:18:31.610Z] ] *** [2026-09-04T17:18:31.610Z] skipping: [an-a01n01] [2026-09-04T17:18:32.796Z] skipping: [an-a01n02] [2026-09-04T17:18:32.796Z] skipping: [an-a01dr01] [2026-09-04T17:18:32.796Z] ok: [an-striker02] [2026-09-04T17:18:32.796Z] ok: [an-striker01] [2026-09-04T17:18:32.796Z] [2026-09-04T17:18:32.796Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-09-04T17:18:32.796Z] ] *** [2026-09-04T17:18:32.796Z] skipping: [an-striker01] [2026-09-04T17:18:32.796Z] skipping: [an-striker02] [2026-09-04T17:18:33.537Z] ok: [an-a01n01] [2026-09-04T17:18:33.537Z] ok: [an-a01n02] [2026-09-04T17:18:33.537Z] ok: [an-a01dr01] [2026-09-04T17:18:33.537Z] [2026-09-04T17:18:33.537Z] TASK [inject failure _raw_params=/usr/bin/false [2026-09-04T17:18:33.537Z] ] ******************************* [2026-09-04T17:18:33.537Z] skipping: [an-striker01] [2026-09-04T17:18:33.537Z] skipping: [an-striker02] [2026-09-04T17:18:33.878Z] skipping: [an-a01n01] [2026-09-04T17:18:33.878Z] skipping: [an-a01n02] [2026-09-04T17:18:33.878Z] skipping: [an-a01dr01] [2026-09-04T17:18:33.878Z] [2026-09-04T17:18:33.878Z] PLAY [Record anvil available resources from striker01] ************************* [2026-09-04T17:18:33.878Z] [2026-09-04T17:18:33.878Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:18:35.588Z] ok: [an-striker01] [2026-09-04T17:18:35.588Z] [2026-09-04T17:18:35.588Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-09-04T17:18:35.588Z] ] *** [2026-09-04T17:18:39.457Z] ok: [an-striker01] [2026-09-04T17:18:39.457Z] [2026-09-04T17:18:39.457Z] PLAY [check selinux status] **************************************************** [2026-09-04T17:18:39.457Z] [2026-09-04T17:18:39.457Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:18:42.488Z] ok: [an-a01n01] [2026-09-04T17:18:42.488Z] ok: [an-a01n02] [2026-09-04T17:18:42.488Z] ok: [an-a01dr01] [2026-09-04T17:18:42.488Z] ok: [an-striker01] [2026-09-04T17:18:42.488Z] ok: [an-striker02] [2026-09-04T17:18:42.488Z] [2026-09-04T17:18:42.488Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-09-04T17:18:42.488Z] skipping: [an-striker01] [2026-09-04T17:18:42.488Z] skipping: [an-striker02] [2026-09-04T17:18:42.488Z] skipping: [an-a01n01] [2026-09-04T17:18:42.488Z] skipping: [an-a01n02] [2026-09-04T17:18:42.488Z] skipping: [an-a01dr01] [2026-09-04T17:18:42.488Z] [2026-09-04T17:18:42.488Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-09-04T17:18:42.488Z] ok: [an-striker01] => { [2026-09-04T17:18:42.488Z] "ansible_selinux": { [2026-09-04T17:18:42.488Z] "config_mode": "enforcing", [2026-09-04T17:18:42.488Z] "mode": "enforcing", [2026-09-04T17:18:42.488Z] "policyvers": 33, [2026-09-04T17:18:42.488Z] "status": "enabled", [2026-09-04T17:18:42.488Z] "type": "targeted" [2026-09-04T17:18:42.488Z] }, [2026-09-04T17:18:42.488Z] "failed_when_result": false [2026-09-04T17:18:42.488Z] } [2026-09-04T17:18:42.830Z] ok: [an-striker02] => { [2026-09-04T17:18:42.830Z] "ansible_selinux": { [2026-09-04T17:18:42.830Z] "config_mode": "enforcing", [2026-09-04T17:18:42.830Z] "mode": "enforcing", [2026-09-04T17:18:42.830Z] "policyvers": 33, [2026-09-04T17:18:42.830Z] "status": "enabled", [2026-09-04T17:18:42.830Z] "type": "targeted" [2026-09-04T17:18:42.830Z] }, [2026-09-04T17:18:42.830Z] "failed_when_result": false [2026-09-04T17:18:42.830Z] } [2026-09-04T17:18:42.830Z] ok: [an-a01n01] => { [2026-09-04T17:18:42.830Z] "ansible_selinux": { [2026-09-04T17:18:42.830Z] "config_mode": "enforcing", [2026-09-04T17:18:42.830Z] "mode": "enforcing", [2026-09-04T17:18:42.830Z] "policyvers": 33, [2026-09-04T17:18:42.830Z] "status": "enabled", [2026-09-04T17:18:42.830Z] "type": "targeted" [2026-09-04T17:18:42.830Z] }, [2026-09-04T17:18:42.830Z] "failed_when_result": false [2026-09-04T17:18:42.830Z] } [2026-09-04T17:18:42.830Z] ok: [an-a01n02] => { [2026-09-04T17:18:42.830Z] "ansible_selinux": { [2026-09-04T17:18:42.830Z] "config_mode": "enforcing", [2026-09-04T17:18:42.830Z] "mode": "enforcing", [2026-09-04T17:18:42.830Z] "policyvers": 33, [2026-09-04T17:18:42.830Z] "status": "enabled", [2026-09-04T17:18:42.830Z] "type": "targeted" [2026-09-04T17:18:42.830Z] }, [2026-09-04T17:18:42.830Z] "failed_when_result": false [2026-09-04T17:18:42.830Z] } [2026-09-04T17:18:42.830Z] ok: [an-a01dr01] => { [2026-09-04T17:18:42.830Z] "ansible_selinux": { [2026-09-04T17:18:42.830Z] "config_mode": "enforcing", [2026-09-04T17:18:42.830Z] "mode": "enforcing", [2026-09-04T17:18:42.830Z] "policyvers": 33, [2026-09-04T17:18:42.830Z] "status": "enabled", [2026-09-04T17:18:42.830Z] "type": "targeted" [2026-09-04T17:18:42.830Z] }, [2026-09-04T17:18:42.830Z] "failed_when_result": false [2026-09-04T17:18:42.830Z] } [2026-09-04T17:18:42.830Z] [2026-09-04T17:18:42.830Z] PLAY [Create random file on striker01] ***************************************** [2026-09-04T17:18:42.830Z] [2026-09-04T17:18:42.830Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:18:44.541Z] ok: [an-striker01] [2026-09-04T17:18:44.541Z] [2026-09-04T17:18:44.541Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-09-04T17:18:46.861Z] changed: [an-striker01] [2026-09-04T17:18:46.861Z] [2026-09-04T17:18:46.861Z] PLAY [Check archive file propagation] ****************************************** [2026-09-04T17:18:46.861Z] [2026-09-04T17:18:46.861Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:18:49.181Z] ok: [an-a01n02] [2026-09-04T17:18:49.181Z] ok: [an-a01n01] [2026-09-04T17:18:49.181Z] ok: [an-striker02] [2026-09-04T17:18:49.181Z] ok: [an-a01dr01] [2026-09-04T17:18:49.181Z] ok: [an-striker01] [2026-09-04T17:18:49.181Z] [2026-09-04T17:18:49.181Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-09-04T17:18:49.181Z] exit 1 [2026-09-04T17:18:49.181Z] else [2026-09-04T17:18:49.181Z] exit 0 [2026-09-04T17:18:49.181Z] fi [2026-09-04T17:18:49.181Z] ] *** [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-09-04T17:20:12.536Z] ok: [an-striker01] [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-09-04T17:20:12.536Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-09-04T17:20:12.536Z] ok: [an-striker02] [2026-09-04T17:20:12.536Z] ok: [an-a01n01] [2026-09-04T17:20:12.536Z] ok: [an-a01n02] [2026-09-04T17:20:12.536Z] ok: [an-a01dr01] [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] PLAY [test fencing] ************************************************************ [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:20:12.536Z] ok: [an-a01n01] [2026-09-04T17:20:12.536Z] ok: [an-a01n02] [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T17:20:12.536Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T17:20:12.536Z] ] *** [2026-09-04T17:20:12.536Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T17:20:12.536Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T17:20:12.536Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T17:20:12.536Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-09-04T17:20:12.536Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T17:20:12.536Z] ] ********** [2026-09-04T17:20:12.536Z] skipping: [an-a01n02] [2026-09-04T17:20:12.536Z] changed: [an-a01n01] [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T17:20:12.536Z] skipping: [an-a01n02] [2026-09-04T17:20:12.536Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-09-04T17:20:12.536Z] [2026-09-04T17:20:12.536Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T17:23:36.651Z] ok: [an-a01n01 -> localhost] [2026-09-04T17:23:36.651Z] [2026-09-04T17:23:36.651Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T17:23:36.652Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T17:23:36.652Z] [2026-09-04T17:23:36.652Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T17:23:36.652Z] ] *** [2026-09-04T17:23:36.652Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T17:23:36.652Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T17:23:36.652Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T17:23:36.652Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T17:23:36.652Z] [2026-09-04T17:23:36.652Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-09-04T17:23:36.652Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T17:23:36.652Z] [2026-09-04T17:23:36.652Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T17:23:36.652Z] ] ********** [2026-09-04T17:23:36.652Z] skipping: [an-a01n01] [2026-09-04T17:23:36.652Z] changed: [an-a01n02] [2026-09-04T17:23:36.652Z] [2026-09-04T17:23:36.652Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T17:23:36.652Z] skipping: [an-a01n01] [2026-09-04T17:23:36.652Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-09-04T17:23:36.652Z] [2026-09-04T17:23:36.652Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T17:26:45.832Z] ok: [an-a01n02 -> localhost] [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T17:26:45.832Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T17:26:45.832Z] ] *** [2026-09-04T17:26:45.832Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T17:26:45.832Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T17:26:45.832Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T17:26:45.832Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] PLAY [advanced server tests] *************************************************** [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T17:26:45.832Z] ok: [an-a01n01] [2026-09-04T17:26:45.832Z] ok: [an-a01n02] [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T17:26:45.832Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T17:26:45.832Z] [2026-09-04T17:26:45.832Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T17:26:45.832Z] ] *** [2026-09-04T17:26:48.155Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T17:26:48.155Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T17:26:48.155Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T17:26:48.155Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T17:26:48.155Z] [2026-09-04T17:26:48.155Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-09-04T17:26:48.155Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-09-04T17:26:48.155Z] [2026-09-04T17:26:48.155Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-09-04T17:26:48.155Z] if [ -z "$storage" ]; then [2026-09-04T17:26:48.155Z] echo "Unable to detect shared storage" [2026-09-04T17:26:48.155Z] exit 1 [2026-09-04T17:26:48.155Z] fi [2026-09-04T17:26:48.155Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T17:26:48.155Z] ] *** [2026-09-04T17:26:54.178Z] changed: [an-a01n01] [2026-09-04T17:26:54.178Z] changed: [an-a01n02] [2026-09-04T17:26:54.178Z] [2026-09-04T17:26:54.178Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-09-04T17:26:54.178Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-09-04T17:26:54.178Z] [2026-09-04T17:26:54.178Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-09-04T17:26:54.178Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-09-04T17:26:54.178Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-09-04T17:26:54.178Z] echo "Unable to detect all deploy iso" [2026-09-04T17:26:54.178Z] echo "Detected isos" [2026-09-04T17:26:54.178Z] echo "$isos" [2026-09-04T17:26:54.178Z] exit 1 [2026-09-04T17:26:54.178Z] fi [2026-09-04T17:26:54.178Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-09-04T17:26:54.178Z] if [ -z "$installiso" ]; then [2026-09-04T17:26:54.178Z] echo "Unable to detect install iso" [2026-09-04T17:26:54.178Z] echo "Detected isos" [2026-09-04T17:26:54.178Z] echo "$isos" [2026-09-04T17:26:54.178Z] exit 1 [2026-09-04T17:26:54.178Z] fi [2026-09-04T17:26:54.178Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T17:26:54.178Z] ] *** [2026-09-04T17:26:59.033Z] changed: [an-a01n01] [2026-09-04T17:26:59.033Z] changed: [an-a01n02] [2026-09-04T17:26:59.033Z] [2026-09-04T17:26:59.033Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-09-04T17:26:59.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:26:59.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:26:59.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:26:59.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:26:59.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:26:59.375Z] [2026-09-04T17:26:59.375Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T17:26:59.375Z] ] *** [2026-09-04T17:27:01.084Z] skipping: [an-a01n02] [2026-09-04T17:27:01.084Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.930621", "end": "2026-09-04 13:27:00.559381", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 13:26:59.628760", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T17:27:01.084Z] ...ignoring [2026-09-04T17:27:01.084Z] [2026-09-04T17:27:01.084Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T17:27:01.084Z] --ci-test \ [2026-09-04T17:27:01.084Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T17:27:01.084Z] --os almalinux9 \ [2026-09-04T17:27:01.084Z] --cpu 2 --ram 4G \ [2026-09-04T17:27:01.084Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T17:27:01.084Z] --install-media {{ installisouuid }} \ [2026-09-04T17:27:01.084Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T17:27:01.084Z] ] *** [2026-09-04T17:27:10.120Z] skipping: [an-a01n02] [2026-09-04T17:27:10.120Z] changed: [an-a01n01] [2026-09-04T17:27:10.120Z] [2026-09-04T17:27:10.120Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:10.120Z] skipping: [an-a01n01] [2026-09-04T17:27:10.120Z] skipping: [an-a01n02] [2026-09-04T17:27:10.120Z] [2026-09-04T17:27:10.120Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T17:27:10.120Z] skipping: [an-a01n01] [2026-09-04T17:27:10.120Z] skipping: [an-a01n02] [2026-09-04T17:27:10.120Z] [2026-09-04T17:27:10.120Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T17:27:10.120Z] skipping: [an-a01n01] [2026-09-04T17:27:10.120Z] skipping: [an-a01n02] [2026-09-04T17:27:10.120Z] [2026-09-04T17:27:10.120Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T17:27:10.120Z] skipping: [an-a01n01] [2026-09-04T17:27:10.120Z] skipping: [an-a01n02] [2026-09-04T17:27:10.120Z] [2026-09-04T17:27:10.120Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T17:27:10.120Z] ] *** [2026-09-04T17:27:11.308Z] skipping: [an-a01n02] [2026-09-04T17:27:11.308Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.952835", "end": "2026-09-04 13:27:11.188530", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 13:27:10.235695", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T17:27:11.308Z] ...ignoring [2026-09-04T17:27:11.308Z] [2026-09-04T17:27:11.308Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T17:27:11.308Z] --ci-test \ [2026-09-04T17:27:11.308Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T17:27:11.308Z] --os almalinux9 \ [2026-09-04T17:27:11.308Z] --cpu 2 --ram 4G \ [2026-09-04T17:27:11.308Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T17:27:11.308Z] --install-media {{ installisouuid }} \ [2026-09-04T17:27:11.308Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T17:27:11.308Z] ] *** [2026-09-04T17:27:20.343Z] skipping: [an-a01n02] [2026-09-04T17:27:20.343Z] changed: [an-a01n01] [2026-09-04T17:27:20.343Z] [2026-09-04T17:27:20.343Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:20.343Z] skipping: [an-a01n01] [2026-09-04T17:27:20.343Z] skipping: [an-a01n02] [2026-09-04T17:27:20.343Z] [2026-09-04T17:27:20.343Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T17:27:20.343Z] skipping: [an-a01n01] [2026-09-04T17:27:20.343Z] skipping: [an-a01n02] [2026-09-04T17:27:20.343Z] [2026-09-04T17:27:20.343Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T17:27:20.687Z] skipping: [an-a01n01] [2026-09-04T17:27:20.687Z] skipping: [an-a01n02] [2026-09-04T17:27:20.687Z] [2026-09-04T17:27:20.687Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T17:27:20.687Z] skipping: [an-a01n01] [2026-09-04T17:27:20.687Z] skipping: [an-a01n02] [2026-09-04T17:27:20.687Z] [2026-09-04T17:27:20.687Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T17:27:20.687Z] ] *** [2026-09-04T17:27:22.399Z] skipping: [an-a01n02] [2026-09-04T17:27:22.399Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.951664", "end": "2026-09-04 13:27:21.933661", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 13:27:20.981997", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T17:27:22.399Z] ...ignoring [2026-09-04T17:27:22.399Z] [2026-09-04T17:27:22.399Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T17:27:22.399Z] --ci-test \ [2026-09-04T17:27:22.399Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T17:27:22.399Z] --os almalinux9 \ [2026-09-04T17:27:22.399Z] --cpu 2 --ram 4G \ [2026-09-04T17:27:22.399Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T17:27:22.399Z] --install-media {{ installisouuid }} \ [2026-09-04T17:27:22.399Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T17:27:22.399Z] ] *** [2026-09-04T17:27:33.381Z] skipping: [an-a01n02] [2026-09-04T17:27:33.381Z] changed: [an-a01n01] [2026-09-04T17:27:33.381Z] [2026-09-04T17:27:33.381Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:33.381Z] skipping: [an-a01n01] [2026-09-04T17:27:33.382Z] skipping: [an-a01n02] [2026-09-04T17:27:33.382Z] [2026-09-04T17:27:33.382Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T17:27:33.382Z] skipping: [an-a01n01] [2026-09-04T17:27:33.382Z] skipping: [an-a01n02] [2026-09-04T17:27:33.382Z] [2026-09-04T17:27:33.382Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T17:27:33.382Z] skipping: [an-a01n01] [2026-09-04T17:27:33.382Z] skipping: [an-a01n02] [2026-09-04T17:27:33.382Z] [2026-09-04T17:27:33.382Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T17:27:33.382Z] skipping: [an-a01n01] [2026-09-04T17:27:33.382Z] skipping: [an-a01n02] [2026-09-04T17:27:33.382Z] [2026-09-04T17:27:33.382Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T17:27:33.382Z] ] *** [2026-09-04T17:27:34.116Z] skipping: [an-a01n02] [2026-09-04T17:27:34.116Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.972503", "end": "2026-09-04 13:27:33.716879", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 13:27:32.744376", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T17:27:34.116Z] ...ignoring [2026-09-04T17:27:34.116Z] [2026-09-04T17:27:34.116Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T17:27:34.116Z] --ci-test \ [2026-09-04T17:27:34.116Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T17:27:34.116Z] --os almalinux9 \ [2026-09-04T17:27:34.116Z] --cpu 2 --ram 4G \ [2026-09-04T17:27:34.116Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T17:27:34.116Z] --install-media {{ installisouuid }} \ [2026-09-04T17:27:34.116Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T17:27:34.116Z] ] *** [2026-09-04T17:27:43.149Z] skipping: [an-a01n02] [2026-09-04T17:27:43.149Z] changed: [an-a01n01] [2026-09-04T17:27:43.149Z] [2026-09-04T17:27:43.149Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:43.149Z] skipping: [an-a01n01] [2026-09-04T17:27:43.149Z] skipping: [an-a01n02] [2026-09-04T17:27:43.149Z] [2026-09-04T17:27:43.149Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T17:27:43.149Z] skipping: [an-a01n01] [2026-09-04T17:27:43.149Z] skipping: [an-a01n02] [2026-09-04T17:27:43.149Z] [2026-09-04T17:27:43.149Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T17:27:43.149Z] skipping: [an-a01n01] [2026-09-04T17:27:43.149Z] skipping: [an-a01n02] [2026-09-04T17:27:43.149Z] [2026-09-04T17:27:43.149Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T17:27:43.149Z] skipping: [an-a01n01] [2026-09-04T17:27:43.149Z] skipping: [an-a01n02] [2026-09-04T17:27:43.149Z] [2026-09-04T17:27:43.149Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T17:27:43.149Z] ] *** [2026-09-04T17:27:44.862Z] skipping: [an-a01n02] [2026-09-04T17:27:44.862Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.026777", "end": "2026-09-04 13:27:44.600598", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 13:27:43.573821", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T17:27:44.862Z] ...ignoring [2026-09-04T17:27:44.862Z] [2026-09-04T17:27:44.862Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T17:27:44.862Z] --ci-test \ [2026-09-04T17:27:44.862Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T17:27:44.862Z] --os almalinux9 \ [2026-09-04T17:27:44.862Z] --cpu 2 --ram 4G \ [2026-09-04T17:27:44.862Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T17:27:44.862Z] --install-media {{ installisouuid }} \ [2026-09-04T17:27:44.862Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T17:27:44.862Z] ] *** [2026-09-04T17:27:55.989Z] skipping: [an-a01n02] [2026-09-04T17:27:55.989Z] changed: [an-a01n01] [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:55.989Z] skipping: [an-a01n01] [2026-09-04T17:27:55.989Z] skipping: [an-a01n02] [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T17:27:55.989Z] skipping: [an-a01n01] [2026-09-04T17:27:55.989Z] skipping: [an-a01n02] [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T17:27:55.989Z] skipping: [an-a01n01] [2026-09-04T17:27:55.989Z] skipping: [an-a01n02] [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T17:27:55.989Z] skipping: [an-a01n01] [2026-09-04T17:27:55.989Z] skipping: [an-a01n02] [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-09-04T17:27:55.989Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:27:55.989Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:27:55.989Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:27:55.989Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:27:55.989Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:27:55.989Z] [2026-09-04T17:27:55.989Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T17:27:55.989Z] ] *** [2026-09-04T17:28:23.956Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-09-04T17:28:23.956Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-09-04T17:28:23.956Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-09-04T17:28:23.956Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-09-04T17:28:23.956Z] ok: [an-a01n01] [2026-09-04T17:28:23.956Z] ok: [an-a01n02] [2026-09-04T17:28:23.956Z] [2026-09-04T17:28:23.956Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T17:28:23.956Z] ] *** [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-09-04T17:30:32.592Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-09-04T17:30:32.592Z] ok: [an-a01n01] [2026-09-04T17:30:32.592Z] ok: [an-a01n02] [2026-09-04T17:30:32.592Z] [2026-09-04T17:30:32.592Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T17:30:32.592Z] ] *** [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-09-04T17:33:11.430Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-09-04T17:33:11.430Z] ok: [an-a01n02] [2026-09-04T17:33:11.430Z] ok: [an-a01n01] [2026-09-04T17:33:11.430Z] [2026-09-04T17:33:11.430Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T17:33:11.430Z] ] *** [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-09-04T17:36:05.393Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-09-04T17:36:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-09-04T17:36:05.394Z] ok: [an-a01n02] [2026-09-04T17:36:05.394Z] ok: [an-a01n01] [2026-09-04T17:36:05.394Z] [2026-09-04T17:36:05.394Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T17:36:05.394Z] ] *** [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-09-04T17:39:14.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-09-04T17:39:14.419Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-09-04T17:39:14.419Z] ok: [an-a01n02] [2026-09-04T17:39:14.419Z] ok: [an-a01n01] [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-09-04T17:39:14.419Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:39:14.419Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:39:14.419Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:39:14.419Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:39:14.419Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T17:39:14.419Z] ] *** [2026-09-04T17:39:14.419Z] ok: [an-a01n02] [2026-09-04T17:39:14.419Z] ok: [an-a01n01] [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T17:39:14.419Z] ] *** [2026-09-04T17:39:14.419Z] ok: [an-a01n02] [2026-09-04T17:39:14.419Z] ok: [an-a01n01] [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T17:39:14.419Z] ] *** [2026-09-04T17:39:14.419Z] ok: [an-a01n01] [2026-09-04T17:39:14.419Z] ok: [an-a01n02] [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T17:39:14.419Z] ] *** [2026-09-04T17:39:14.419Z] ok: [an-a01n02] [2026-09-04T17:39:14.419Z] ok: [an-a01n01] [2026-09-04T17:39:14.419Z] [2026-09-04T17:39:14.419Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T17:39:14.419Z] ] *** [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (351 retries left). [2026-09-04T17:41:07.961Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (351 retries left). [2026-09-04T17:41:07.961Z] ok: [an-a01n02] [2026-09-04T17:41:07.961Z] ok: [an-a01n01] [2026-09-04T17:41:07.961Z] [2026-09-04T17:41:07.961Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-09-04T17:41:07.961Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:41:07.961Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:41:07.961Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:41:07.961Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:41:07.961Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:41:07.961Z] [2026-09-04T17:41:07.961Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T17:41:07.961Z] ] *** [2026-09-04T17:41:07.961Z] ok: [an-a01n02] [2026-09-04T17:41:07.961Z] ok: [an-a01n01] [2026-09-04T17:41:07.961Z] [2026-09-04T17:41:07.961Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T17:41:07.961Z] ] *** [2026-09-04T17:41:07.961Z] ok: [an-a01n02] [2026-09-04T17:41:07.961Z] ok: [an-a01n01] [2026-09-04T17:41:07.961Z] [2026-09-04T17:41:07.961Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T17:41:07.961Z] ] *** [2026-09-04T17:41:13.982Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-09-04T17:41:13.982Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-09-04T17:41:13.982Z] ok: [an-a01n02] [2026-09-04T17:41:13.982Z] ok: [an-a01n01] [2026-09-04T17:41:13.982Z] [2026-09-04T17:41:13.982Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T17:41:13.982Z] ] *** [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-09-04T17:44:07.928Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-09-04T17:44:07.928Z] ok: [an-a01n01] [2026-09-04T17:44:07.928Z] ok: [an-a01n02] [2026-09-04T17:44:07.928Z] [2026-09-04T17:44:07.928Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T17:44:07.928Z] ] *** [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left). [2026-09-04T17:48:17.498Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left). [2026-09-04T17:48:17.498Z] ok: [an-a01n01] [2026-09-04T17:48:17.498Z] ok: [an-a01n02] [2026-09-04T17:48:17.498Z] [2026-09-04T17:48:17.498Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-09-04T17:48:17.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:48:17.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:48:17.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:48:17.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:48:17.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:48:17.498Z] [2026-09-04T17:48:17.498Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T17:48:17.498Z] ] *** [2026-09-04T17:48:19.841Z] ok: [an-a01n01] [2026-09-04T17:48:19.841Z] ok: [an-a01n02] [2026-09-04T17:48:19.841Z] [2026-09-04T17:48:19.841Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T17:48:19.841Z] ] *** [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-09-04T17:49:29.391Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-09-04T17:49:29.391Z] ok: [an-a01n02] [2026-09-04T17:49:29.391Z] ok: [an-a01n01] [2026-09-04T17:49:29.391Z] [2026-09-04T17:49:29.391Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T17:49:29.391Z] ] *** [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-09-04T17:52:23.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-09-04T17:52:23.310Z] ok: [an-a01n02] [2026-09-04T17:52:23.310Z] ok: [an-a01n01] [2026-09-04T17:52:23.310Z] [2026-09-04T17:52:23.310Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T17:52:23.310Z] ] *** [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-09-04T17:54:17.012Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-09-04T17:54:17.012Z] ok: [an-a01n02] [2026-09-04T17:54:17.012Z] ok: [an-a01n01] [2026-09-04T17:54:17.012Z] [2026-09-04T17:54:17.012Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T17:54:17.012Z] ] *** [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left). [2026-09-04T17:57:26.008Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left). [2026-09-04T17:57:26.008Z] ok: [an-a01n02] [2026-09-04T17:57:26.008Z] ok: [an-a01n01] [2026-09-04T17:57:26.008Z] [2026-09-04T17:57:26.008Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-09-04T17:57:26.008Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T17:57:26.008Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T17:57:26.008Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T17:57:26.008Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T17:57:26.008Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T17:57:26.008Z] [2026-09-04T17:57:26.008Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-09-04T17:57:26.008Z] ok: [an-a01n01] [2026-09-04T17:57:26.008Z] ok: [an-a01n02] [2026-09-04T17:57:26.008Z] [2026-09-04T17:57:26.009Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T17:57:26.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T17:57:26.009Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T17:57:26.009Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T17:57:26.009Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T17:57:26.009Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T17:57:26.009Z] echo "pcs:" [2026-09-04T17:57:26.009Z] echo "$pcsstatus" [2026-09-04T17:57:26.009Z] exit 1 [2026-09-04T17:57:26.009Z] fi [2026-09-04T17:57:26.009Z] exit 0 [2026-09-04T17:57:26.009Z] ] *** [2026-09-04T17:57:26.009Z] ok: [an-a01n01] [2026-09-04T17:57:26.009Z] ok: [an-a01n02] [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T17:57:26.009Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T17:57:26.009Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T17:57:26.009Z] ret=0 [2026-09-04T17:57:26.009Z] if [ "$servers" != "1" ]; then [2026-09-04T17:57:26.009Z] echo "Not all servers are running on the same node!" [2026-09-04T17:57:26.009Z] echo "$pcsstatus" [2026-09-04T17:57:26.009Z] ret=1 [2026-09-04T17:57:26.009Z] fi [2026-09-04T17:57:26.009Z] echo "$hypervisor" [2026-09-04T17:57:26.009Z] exit $ret [2026-09-04T17:57:26.009Z] ] *** [2026-09-04T17:57:26.009Z] ok: [an-a01n01] [2026-09-04T17:57:26.009Z] ok: [an-a01n02] [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-09-04T17:57:26.009Z] ok: [an-a01n01] [2026-09-04T17:57:26.009Z] ok: [an-a01n02] [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-09-04T17:57:26.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T17:57:26.009Z] ] ********** [2026-09-04T17:57:26.009Z] skipping: [an-a01n02] [2026-09-04T17:57:26.009Z] changed: [an-a01n01] [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T17:57:26.009Z] skipping: [an-a01n01] [2026-09-04T17:57:26.009Z] skipping: [an-a01n02] [2026-09-04T17:57:26.009Z] [2026-09-04T17:57:26.009Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-09-04T17:57:26.349Z] skipping: [an-a01n01] [2026-09-04T17:57:26.349Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-09-04T17:57:26.349Z] [2026-09-04T17:57:26.349Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-09-04T17:57:26.349Z] ] *** [2026-09-04T17:58:35.888Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-09-04T17:58:35.888Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-09-04T17:58:35.888Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-09-04T17:58:35.888Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-09-04T17:58:35.888Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-09-04T17:58:35.888Z] ok: [an-a01n02] [2026-09-04T17:58:35.888Z] [2026-09-04T17:58:35.888Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T17:58:35.888Z] skipping: [an-a01n01] [2026-09-04T17:58:35.888Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-09-04T17:58:35.888Z] [2026-09-04T17:58:35.888Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T17:58:35.888Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T17:58:35.888Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T17:58:35.888Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T17:58:35.888Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T17:58:35.888Z] echo "pcs:" [2026-09-04T17:58:35.888Z] echo "$pcsstatus" [2026-09-04T17:58:35.888Z] exit 1 [2026-09-04T17:58:35.888Z] fi [2026-09-04T17:58:35.888Z] exit 0 [2026-09-04T17:58:35.888Z] ] *** [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-09-04T18:00:29.436Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (112 retries left). [2026-09-04T18:00:29.436Z] ok: [an-a01n02] [2026-09-04T18:00:29.436Z] [2026-09-04T18:00:29.436Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:00:29.436Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:00:29.436Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:00:29.436Z] ret=0 [2026-09-04T18:00:29.437Z] if [ "$servers" != "1" ]; then [2026-09-04T18:00:29.437Z] echo "Not all servers are running on the same node!" [2026-09-04T18:00:29.437Z] echo "$pcsstatus" [2026-09-04T18:00:29.437Z] ret=1 [2026-09-04T18:00:29.437Z] fi [2026-09-04T18:00:29.437Z] echo "$hypervisor" [2026-09-04T18:00:29.437Z] exit $ret [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.437Z] ok: [an-a01n02] [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] => (item=1) [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] => (item=2) [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] => (item=3) [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] => (item=4) [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] => (item=5) [2026-09-04T18:00:29.437Z] skipping: [an-a01n01] [2026-09-04T18:00:29.437Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-09-04T18:00:29.437Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-09-04T18:00:29.437Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-09-04T18:00:29.437Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-09-04T18:00:29.437Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.437Z] ok: [an-a01n02] [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.437Z] ok: [an-a01n02] [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.437Z] ok: [an-a01n02] [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.437Z] ok: [an-a01n02] [2026-09-04T18:00:29.437Z] [2026-09-04T18:00:29.437Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:00:29.437Z] ] *** [2026-09-04T18:00:29.779Z] ok: [an-a01n02] [2026-09-04T18:00:29.779Z] [2026-09-04T18:00:29.779Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-09-04T18:00:29.779Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:00:29.779Z] [2026-09-04T18:00:29.779Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T18:03:54.076Z] ok: [an-a01n01 -> localhost] [2026-09-04T18:03:54.076Z] ok: [an-a01n02 -> localhost] [2026-09-04T18:03:54.076Z] [2026-09-04T18:03:54.076Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:03:54.076Z] [2026-09-04T18:03:54.076Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:03:54.076Z] ] *** [2026-09-04T18:03:54.076Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:03:54.076Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:03:54.076Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:03:54.076Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:03:54.076Z] [2026-09-04T18:03:54.076Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T18:03:54.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T18:03:54.076Z] [2026-09-04T18:03:54.076Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:03:54.076Z] ] *** [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-09-04T18:07:03.076Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:07:03.076Z] ] *** [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:07:03.076Z] ] *** [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:07:03.076Z] ] *** [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:07:03.076Z] ] *** [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-09-04T18:07:03.076Z] ok: [an-a01n01] [2026-09-04T18:07:03.076Z] ok: [an-a01n02] [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:07:03.076Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:07:03.076Z] [2026-09-04T18:07:03.076Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:07:03.076Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:07:03.076Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:07:03.076Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:07:03.076Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:07:03.076Z] echo "pcs:" [2026-09-04T18:07:03.076Z] echo "$pcsstatus" [2026-09-04T18:07:03.076Z] exit 1 [2026-09-04T18:07:03.076Z] fi [2026-09-04T18:07:03.076Z] exit 0 [2026-09-04T18:07:03.076Z] ] *** [2026-09-04T18:07:05.400Z] ok: [an-a01n01] [2026-09-04T18:07:05.400Z] ok: [an-a01n02] [2026-09-04T18:07:05.400Z] [2026-09-04T18:07:05.400Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:07:05.400Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:07:05.400Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:07:05.400Z] ret=0 [2026-09-04T18:07:05.400Z] if [ "$servers" != "1" ]; then [2026-09-04T18:07:05.400Z] echo "Not all servers are running on the same node!" [2026-09-04T18:07:05.400Z] echo "$pcsstatus" [2026-09-04T18:07:05.400Z] ret=1 [2026-09-04T18:07:05.400Z] fi [2026-09-04T18:07:05.400Z] echo "$hypervisor" [2026-09-04T18:07:05.400Z] exit $ret [2026-09-04T18:07:05.400Z] ] *** [2026-09-04T18:07:07.723Z] ok: [an-a01n01] [2026-09-04T18:07:07.723Z] ok: [an-a01n02] [2026-09-04T18:07:07.723Z] [2026-09-04T18:07:07.723Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-09-04T18:07:07.723Z] ok: [an-a01n01] [2026-09-04T18:07:07.723Z] ok: [an-a01n02] [2026-09-04T18:07:07.723Z] [2026-09-04T18:07:07.723Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-09-04T18:07:07.723Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:07:07.723Z] [2026-09-04T18:07:07.723Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T18:07:07.723Z] ] ********** [2026-09-04T18:07:08.456Z] skipping: [an-a01n01] [2026-09-04T18:07:08.456Z] changed: [an-a01n02] [2026-09-04T18:07:08.456Z] [2026-09-04T18:07:08.456Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T18:07:08.456Z] skipping: [an-a01n01] [2026-09-04T18:07:08.456Z] skipping: [an-a01n02] [2026-09-04T18:07:08.456Z] [2026-09-04T18:07:08.456Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-09-04T18:07:08.456Z] skipping: [an-a01n02] [2026-09-04T18:07:08.456Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-09-04T18:07:08.456Z] [2026-09-04T18:07:08.456Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-09-04T18:07:08.456Z] ] *** [2026-09-04T18:08:17.975Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-09-04T18:08:17.975Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-09-04T18:08:17.975Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-09-04T18:08:17.975Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-09-04T18:08:17.975Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-09-04T18:08:17.975Z] ok: [an-a01n01] [2026-09-04T18:08:17.975Z] [2026-09-04T18:08:17.975Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:08:17.975Z] skipping: [an-a01n02] [2026-09-04T18:08:17.975Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-09-04T18:08:17.975Z] [2026-09-04T18:08:17.975Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:08:17.975Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:08:17.975Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:08:17.975Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:08:17.975Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:08:17.975Z] echo "pcs:" [2026-09-04T18:08:17.975Z] echo "$pcsstatus" [2026-09-04T18:08:17.975Z] exit 1 [2026-09-04T18:08:17.975Z] fi [2026-09-04T18:08:17.975Z] exit 0 [2026-09-04T18:08:17.975Z] ] *** [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-09-04T18:09:27.521Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-09-04T18:09:27.521Z] ok: [an-a01n01] [2026-09-04T18:09:27.521Z] [2026-09-04T18:09:27.521Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:09:27.521Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:09:27.521Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:09:27.521Z] ret=0 [2026-09-04T18:09:27.521Z] if [ "$servers" != "1" ]; then [2026-09-04T18:09:27.521Z] echo "Not all servers are running on the same node!" [2026-09-04T18:09:27.521Z] echo "$pcsstatus" [2026-09-04T18:09:27.521Z] ret=1 [2026-09-04T18:09:27.521Z] fi [2026-09-04T18:09:27.521Z] echo "$hypervisor" [2026-09-04T18:09:27.521Z] exit $ret [2026-09-04T18:09:27.521Z] ] *** [2026-09-04T18:09:29.852Z] ok: [an-a01n01] [2026-09-04T18:09:29.852Z] [2026-09-04T18:09:29.852Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] => (item=1) [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] => (item=2) [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] => (item=3) [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] => (item=4) [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] => (item=5) [2026-09-04T18:09:30.194Z] skipping: [an-a01n02] [2026-09-04T18:09:30.194Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-09-04T18:09:30.194Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-09-04T18:09:30.194Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-09-04T18:09:30.194Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-09-04T18:09:30.194Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-09-04T18:09:30.194Z] [2026-09-04T18:09:30.194Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:09:30.194Z] ] *** [2026-09-04T18:10:18.721Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-09-04T18:10:18.721Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (59 retries left). [2026-09-04T18:10:18.721Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (58 retries left). [2026-09-04T18:10:18.721Z] ok: [an-a01n01] [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:10:18.721Z] ] *** [2026-09-04T18:10:18.721Z] ok: [an-a01n01] [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:10:18.721Z] ] *** [2026-09-04T18:10:18.721Z] ok: [an-a01n01] [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:10:18.721Z] ] *** [2026-09-04T18:10:18.721Z] ok: [an-a01n01] [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:10:18.721Z] ] *** [2026-09-04T18:10:18.721Z] ok: [an-a01n01] [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-09-04T18:10:18.721Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:10:18.721Z] [2026-09-04T18:10:18.721Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T18:13:27.730Z] ok: [an-a01n01 -> localhost] [2026-09-04T18:13:27.730Z] ok: [an-a01n02 -> localhost] [2026-09-04T18:13:27.730Z] [2026-09-04T18:13:27.730Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-09-04T18:13:27.730Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:13:27.730Z] [2026-09-04T18:13:27.730Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:13:27.730Z] ] *** [2026-09-04T18:13:32.603Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:13:32.603Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:13:32.603Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:13:32.603Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:13:32.603Z] [2026-09-04T18:13:32.603Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:13:32.603Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T18:13:32.603Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T18:13:32.603Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T18:13:32.603Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T18:13:32.603Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T18:13:32.603Z] [2026-09-04T18:13:32.603Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:13:32.603Z] ] *** [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-09-04T18:15:26.156Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:15:26.156Z] ] *** [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:15:26.156Z] ] *** [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:15:26.156Z] ] *** [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:15:26.156Z] ] *** [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-09-04T18:15:26.156Z] ok: [an-a01n01] [2026-09-04T18:15:26.156Z] ok: [an-a01n02] [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:15:26.156Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:15:26.156Z] [2026-09-04T18:15:26.156Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:15:26.156Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:15:26.156Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:15:26.156Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:15:26.156Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:15:26.156Z] echo "pcs:" [2026-09-04T18:15:26.156Z] echo "$pcsstatus" [2026-09-04T18:15:26.156Z] exit 1 [2026-09-04T18:15:26.156Z] fi [2026-09-04T18:15:26.156Z] exit 0 [2026-09-04T18:15:26.156Z] ] *** [2026-09-04T18:15:27.867Z] ok: [an-a01n02] [2026-09-04T18:15:27.867Z] ok: [an-a01n01] [2026-09-04T18:15:27.867Z] [2026-09-04T18:15:27.867Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:15:27.867Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:15:27.867Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:15:27.867Z] ret=0 [2026-09-04T18:15:27.867Z] if [ "$servers" != "1" ]; then [2026-09-04T18:15:27.867Z] echo "Not all servers are running on the same node!" [2026-09-04T18:15:27.867Z] echo "$pcsstatus" [2026-09-04T18:15:27.867Z] ret=1 [2026-09-04T18:15:27.867Z] fi [2026-09-04T18:15:27.867Z] echo "$hypervisor" [2026-09-04T18:15:27.867Z] exit $ret [2026-09-04T18:15:27.867Z] ] *** [2026-09-04T18:15:29.576Z] ok: [an-a01n01] [2026-09-04T18:15:29.576Z] ok: [an-a01n02] [2026-09-04T18:15:29.576Z] [2026-09-04T18:15:29.576Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-09-04T18:15:29.576Z] ok: [an-a01n01] [2026-09-04T18:15:29.576Z] ok: [an-a01n02] [2026-09-04T18:15:29.576Z] [2026-09-04T18:15:29.576Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-09-04T18:15:29.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:15:29.918Z] [2026-09-04T18:15:29.918Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T18:15:29.918Z] ] ********** [2026-09-04T18:15:30.651Z] skipping: [an-a01n02] [2026-09-04T18:15:30.651Z] changed: [an-a01n01] [2026-09-04T18:15:30.651Z] [2026-09-04T18:15:30.651Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T18:15:30.651Z] skipping: [an-a01n01] [2026-09-04T18:15:30.651Z] skipping: [an-a01n02] [2026-09-04T18:15:30.651Z] [2026-09-04T18:15:30.651Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-09-04T18:15:30.651Z] skipping: [an-a01n01] [2026-09-04T18:15:30.651Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-09-04T18:15:30.651Z] [2026-09-04T18:15:30.651Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-09-04T18:15:30.651Z] ] *** [2026-09-04T18:16:40.358Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-09-04T18:16:40.358Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-09-04T18:16:40.358Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-09-04T18:16:40.358Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-09-04T18:16:40.358Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-09-04T18:16:40.358Z] ok: [an-a01n02] [2026-09-04T18:16:40.358Z] [2026-09-04T18:16:40.358Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:16:40.358Z] skipping: [an-a01n01] [2026-09-04T18:16:40.358Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-09-04T18:16:40.358Z] [2026-09-04T18:16:40.358Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:16:40.358Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:16:40.358Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:16:40.358Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:16:40.358Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:16:40.358Z] echo "pcs:" [2026-09-04T18:16:40.358Z] echo "$pcsstatus" [2026-09-04T18:16:40.358Z] exit 1 [2026-09-04T18:16:40.358Z] fi [2026-09-04T18:16:40.358Z] exit 0 [2026-09-04T18:16:40.358Z] ] *** [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-09-04T18:17:49.877Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-09-04T18:17:49.877Z] ok: [an-a01n02] [2026-09-04T18:17:49.877Z] [2026-09-04T18:17:49.877Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:17:49.877Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:17:49.877Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:17:49.877Z] ret=0 [2026-09-04T18:17:49.877Z] if [ "$servers" != "1" ]; then [2026-09-04T18:17:49.877Z] echo "Not all servers are running on the same node!" [2026-09-04T18:17:49.877Z] echo "$pcsstatus" [2026-09-04T18:17:49.877Z] ret=1 [2026-09-04T18:17:49.877Z] fi [2026-09-04T18:17:49.877Z] echo "$hypervisor" [2026-09-04T18:17:49.877Z] exit $ret [2026-09-04T18:17:49.877Z] ] *** [2026-09-04T18:17:52.199Z] ok: [an-a01n02] [2026-09-04T18:17:52.199Z] [2026-09-04T18:17:52.199Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] => (item=1) [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] => (item=2) [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] => (item=3) [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] => (item=4) [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] => (item=5) [2026-09-04T18:17:52.199Z] skipping: [an-a01n01] [2026-09-04T18:17:52.199Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-09-04T18:17:52.199Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-09-04T18:17:52.199Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-09-04T18:17:52.199Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-09-04T18:17:52.199Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-09-04T18:17:52.199Z] [2026-09-04T18:17:52.199Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:17:52.199Z] ] *** [2026-09-04T18:17:52.934Z] ok: [an-a01n02] [2026-09-04T18:17:52.934Z] [2026-09-04T18:17:52.934Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:17:52.934Z] ] *** [2026-09-04T18:17:53.668Z] ok: [an-a01n02] [2026-09-04T18:17:53.668Z] [2026-09-04T18:17:53.668Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:17:53.668Z] ] *** [2026-09-04T18:17:54.854Z] ok: [an-a01n02] [2026-09-04T18:17:54.854Z] [2026-09-04T18:17:54.854Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:17:54.854Z] ] *** [2026-09-04T18:17:55.587Z] ok: [an-a01n02] [2026-09-04T18:17:55.587Z] [2026-09-04T18:17:55.587Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:17:55.587Z] ] *** [2026-09-04T18:17:56.320Z] ok: [an-a01n02] [2026-09-04T18:17:56.320Z] [2026-09-04T18:17:56.320Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-09-04T18:17:56.320Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:17:56.320Z] [2026-09-04T18:17:56.320Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n01 -> localhost] [2026-09-04T18:21:20.460Z] ok: [an-a01n02 -> localhost] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:21:20.460Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:21:20.460Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:21:20.460Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=1) [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=2) [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=3) [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=4) [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=5) [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:21:20.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:21:20.460Z] [2026-09-04T18:21:20.460Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:21:20.460Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:21:20.460Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:21:20.460Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:21:20.460Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:21:20.460Z] echo "pcs:" [2026-09-04T18:21:20.460Z] echo "$pcsstatus" [2026-09-04T18:21:20.460Z] exit 1 [2026-09-04T18:21:20.460Z] fi [2026-09-04T18:21:20.460Z] exit 0 [2026-09-04T18:21:20.460Z] ] *** [2026-09-04T18:21:20.460Z] ok: [an-a01n02] [2026-09-04T18:21:20.460Z] ok: [an-a01n01] [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:21:20.461Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:21:20.461Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:21:20.461Z] ret=0 [2026-09-04T18:21:20.461Z] if [ "$servers" != "1" ]; then [2026-09-04T18:21:20.461Z] echo "Not all servers are running on the same node!" [2026-09-04T18:21:20.461Z] echo "$pcsstatus" [2026-09-04T18:21:20.461Z] ret=1 [2026-09-04T18:21:20.461Z] fi [2026-09-04T18:21:20.461Z] echo "$hypervisor" [2026-09-04T18:21:20.461Z] exit $ret [2026-09-04T18:21:20.461Z] ] *** [2026-09-04T18:21:20.461Z] ok: [an-a01n02] [2026-09-04T18:21:20.461Z] ok: [an-a01n01] [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-09-04T18:21:20.461Z] ok: [an-a01n01] [2026-09-04T18:21:20.461Z] ok: [an-a01n02] [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-09-04T18:21:20.461Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T18:21:20.461Z] ] ********** [2026-09-04T18:21:20.461Z] skipping: [an-a01n01] [2026-09-04T18:21:20.461Z] changed: [an-a01n02] [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T18:21:20.461Z] skipping: [an-a01n01] [2026-09-04T18:21:20.461Z] skipping: [an-a01n02] [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-09-04T18:21:20.461Z] skipping: [an-a01n02] [2026-09-04T18:21:20.461Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-09-04T18:21:20.461Z] [2026-09-04T18:21:20.461Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-09-04T18:21:20.461Z] ] *** [2026-09-04T18:22:29.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-09-04T18:22:29.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-09-04T18:22:29.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-09-04T18:22:29.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-09-04T18:22:29.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-09-04T18:22:29.980Z] ok: [an-a01n01] [2026-09-04T18:22:29.980Z] [2026-09-04T18:22:29.980Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:22:29.980Z] skipping: [an-a01n02] [2026-09-04T18:22:29.980Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-09-04T18:22:29.980Z] [2026-09-04T18:22:29.980Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:22:29.981Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:22:29.981Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:22:29.981Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:22:29.981Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:22:29.981Z] echo "pcs:" [2026-09-04T18:22:29.981Z] echo "$pcsstatus" [2026-09-04T18:22:29.981Z] exit 1 [2026-09-04T18:22:29.981Z] fi [2026-09-04T18:22:29.981Z] exit 0 [2026-09-04T18:22:29.981Z] ] *** [2026-09-04T18:23:28.239Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-09-04T18:23:28.239Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-09-04T18:23:28.239Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-09-04T18:23:28.239Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-09-04T18:23:28.239Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-09-04T18:23:28.239Z] ok: [an-a01n01] [2026-09-04T18:23:28.239Z] [2026-09-04T18:23:28.239Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:23:28.239Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:23:28.239Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:23:28.239Z] ret=0 [2026-09-04T18:23:28.239Z] if [ "$servers" != "1" ]; then [2026-09-04T18:23:28.239Z] echo "Not all servers are running on the same node!" [2026-09-04T18:23:28.239Z] echo "$pcsstatus" [2026-09-04T18:23:28.239Z] ret=1 [2026-09-04T18:23:28.239Z] fi [2026-09-04T18:23:28.239Z] echo "$hypervisor" [2026-09-04T18:23:28.239Z] exit $ret [2026-09-04T18:23:28.239Z] ] *** [2026-09-04T18:23:28.239Z] ok: [an-a01n01] [2026-09-04T18:23:28.239Z] [2026-09-04T18:23:28.239Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] => (item=1) [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] => (item=2) [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] => (item=3) [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] => (item=4) [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] => (item=5) [2026-09-04T18:23:28.239Z] skipping: [an-a01n02] [2026-09-04T18:23:28.239Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-09-04T18:23:28.239Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-09-04T18:23:28.239Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-09-04T18:23:28.239Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-09-04T18:23:28.239Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-09-04T18:23:28.239Z] [2026-09-04T18:23:28.239Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:23:28.239Z] ] *** [2026-09-04T18:23:32.114Z] ok: [an-a01n01] [2026-09-04T18:23:32.114Z] [2026-09-04T18:23:32.114Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:23:32.114Z] ] *** [2026-09-04T18:23:48.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-09-04T18:23:48.197Z] ok: [an-a01n01] [2026-09-04T18:23:48.197Z] [2026-09-04T18:23:48.197Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:23:48.197Z] ] *** [2026-09-04T18:23:48.197Z] ok: [an-a01n01] [2026-09-04T18:23:48.197Z] [2026-09-04T18:23:48.197Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:23:48.197Z] ] *** [2026-09-04T18:23:48.197Z] ok: [an-a01n01] [2026-09-04T18:23:48.197Z] [2026-09-04T18:23:48.197Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:23:48.197Z] ] *** [2026-09-04T18:23:48.931Z] ok: [an-a01n01] [2026-09-04T18:23:48.931Z] [2026-09-04T18:23:48.931Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-09-04T18:23:49.273Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:23:49.273Z] [2026-09-04T18:23:49.273Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01 -> localhost] [2026-09-04T18:27:13.367Z] ok: [an-a01n02 -> localhost] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:27:13.367Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:27:13.367Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:27:13.367Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:27:13.367Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:27:13.367Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:27:13.367Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:27:13.367Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:27:13.367Z] echo "pcs:" [2026-09-04T18:27:13.367Z] echo "$pcsstatus" [2026-09-04T18:27:13.367Z] exit 1 [2026-09-04T18:27:13.367Z] fi [2026-09-04T18:27:13.367Z] exit 0 [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:27:13.367Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:27:13.367Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:27:13.367Z] ret=0 [2026-09-04T18:27:13.367Z] if [ "$servers" != "1" ]; then [2026-09-04T18:27:13.367Z] echo "Not all servers are running on the same node!" [2026-09-04T18:27:13.367Z] echo "$pcsstatus" [2026-09-04T18:27:13.367Z] ret=1 [2026-09-04T18:27:13.367Z] fi [2026-09-04T18:27:13.367Z] echo "$hypervisor" [2026-09-04T18:27:13.367Z] exit $ret [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-09-04T18:27:13.367Z] ok: [an-a01n01] [2026-09-04T18:27:13.367Z] ok: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-09-04T18:27:13.367Z] ] ********** [2026-09-04T18:27:13.367Z] skipping: [an-a01n02] [2026-09-04T18:27:13.367Z] changed: [an-a01n01] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-09-04T18:27:13.367Z] skipping: [an-a01n01] [2026-09-04T18:27:13.367Z] skipping: [an-a01n02] [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-09-04T18:27:13.367Z] skipping: [an-a01n01] [2026-09-04T18:27:13.367Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-09-04T18:27:13.367Z] [2026-09-04T18:27:13.367Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-09-04T18:27:13.367Z] ] *** [2026-09-04T18:28:22.887Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-09-04T18:28:22.887Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-09-04T18:28:22.887Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-09-04T18:28:22.887Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-09-04T18:28:22.887Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-09-04T18:28:22.887Z] ok: [an-a01n02] [2026-09-04T18:28:22.887Z] [2026-09-04T18:28:22.887Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-09-04T18:28:22.887Z] skipping: [an-a01n01] [2026-09-04T18:28:22.887Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-09-04T18:28:22.887Z] [2026-09-04T18:28:22.887Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:28:22.887Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:28:22.887Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:28:22.887Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:28:22.887Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:28:22.887Z] echo "pcs:" [2026-09-04T18:28:22.887Z] echo "$pcsstatus" [2026-09-04T18:28:22.887Z] exit 1 [2026-09-04T18:28:22.887Z] fi [2026-09-04T18:28:22.887Z] exit 0 [2026-09-04T18:28:22.887Z] ] *** [2026-09-04T18:29:21.080Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-09-04T18:29:21.080Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-09-04T18:29:21.080Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-09-04T18:29:21.080Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-09-04T18:29:21.080Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-09-04T18:29:21.080Z] ok: [an-a01n02] [2026-09-04T18:29:21.080Z] [2026-09-04T18:29:21.080Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:29:21.080Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:29:21.080Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:29:21.080Z] ret=0 [2026-09-04T18:29:21.080Z] if [ "$servers" != "1" ]; then [2026-09-04T18:29:21.080Z] echo "Not all servers are running on the same node!" [2026-09-04T18:29:21.080Z] echo "$pcsstatus" [2026-09-04T18:29:21.080Z] ret=1 [2026-09-04T18:29:21.080Z] fi [2026-09-04T18:29:21.080Z] echo "$hypervisor" [2026-09-04T18:29:21.080Z] exit $ret [2026-09-04T18:29:21.080Z] ] *** [2026-09-04T18:29:21.080Z] ok: [an-a01n02] [2026-09-04T18:29:21.080Z] [2026-09-04T18:29:21.080Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] => (item=1) [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] => (item=2) [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] => (item=3) [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] => (item=4) [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] => (item=5) [2026-09-04T18:29:21.080Z] skipping: [an-a01n01] [2026-09-04T18:29:21.080Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-09-04T18:29:21.080Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-09-04T18:29:21.080Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-09-04T18:29:21.080Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-09-04T18:29:21.080Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-09-04T18:29:21.080Z] [2026-09-04T18:29:21.080Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:29:21.080Z] ] *** [2026-09-04T18:29:24.955Z] ok: [an-a01n02] [2026-09-04T18:29:24.955Z] [2026-09-04T18:29:24.955Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:29:24.955Z] ] *** [2026-09-04T18:29:25.297Z] ok: [an-a01n02] [2026-09-04T18:29:25.297Z] [2026-09-04T18:29:25.297Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:29:25.297Z] ] *** [2026-09-04T18:29:41.347Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to respond to ssh (60 retries left). [2026-09-04T18:29:41.347Z] ok: [an-a01n02] [2026-09-04T18:29:41.347Z] [2026-09-04T18:29:41.347Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:29:41.347Z] ] *** [2026-09-04T18:29:41.347Z] ok: [an-a01n02] [2026-09-04T18:29:41.347Z] [2026-09-04T18:29:41.347Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T18:29:41.347Z] ] *** [2026-09-04T18:29:41.347Z] ok: [an-a01n02] [2026-09-04T18:29:41.347Z] [2026-09-04T18:29:41.347Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-09-04T18:29:41.347Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-09-04T18:29:41.347Z] [2026-09-04T18:29:41.347Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n01 -> localhost] [2026-09-04T18:33:05.534Z] ok: [an-a01n02 -> localhost] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:33:05.534Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:33:05.534Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:33:05.534Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] PLAY [basic server tests] ****************************************************** [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:33:05.534Z] ok: [an-a01n01] [2026-09-04T18:33:05.534Z] ok: [an-a01n02] [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T18:33:05.534Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:05.534Z] [2026-09-04T18:33:05.534Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:33:05.534Z] ] *** [2026-09-04T18:33:07.246Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:33:07.246Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:33:07.246Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:33:07.246Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:33:07.246Z] [2026-09-04T18:33:07.246Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-09-04T18:33:07.246Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:07.246Z] [2026-09-04T18:33:07.246Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-09-04T18:33:07.246Z] if [ -z "$storage" ]; then [2026-09-04T18:33:07.246Z] echo "Unable to detect shared storage" [2026-09-04T18:33:07.246Z] exit 1 [2026-09-04T18:33:07.246Z] fi [2026-09-04T18:33:07.246Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:33:07.246Z] ] *** [2026-09-04T18:33:13.266Z] changed: [an-a01n01] [2026-09-04T18:33:13.266Z] changed: [an-a01n02] [2026-09-04T18:33:13.266Z] [2026-09-04T18:33:13.266Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-09-04T18:33:13.266Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:13.266Z] [2026-09-04T18:33:13.266Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-09-04T18:33:13.266Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-09-04T18:33:13.266Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-09-04T18:33:13.266Z] echo "Unable to detect all deploy iso" [2026-09-04T18:33:13.266Z] echo "Detected isos" [2026-09-04T18:33:13.266Z] echo "$isos" [2026-09-04T18:33:13.266Z] exit 1 [2026-09-04T18:33:13.266Z] fi [2026-09-04T18:33:13.266Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-09-04T18:33:13.266Z] if [ -z "$installiso" ]; then [2026-09-04T18:33:13.266Z] echo "Unable to detect install iso" [2026-09-04T18:33:13.266Z] echo "Detected isos" [2026-09-04T18:33:13.266Z] echo "$isos" [2026-09-04T18:33:13.266Z] exit 1 [2026-09-04T18:33:13.266Z] fi [2026-09-04T18:33:13.266Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:33:13.266Z] ] *** [2026-09-04T18:33:19.287Z] changed: [an-a01n01] [2026-09-04T18:33:19.287Z] changed: [an-a01n02] [2026-09-04T18:33:19.287Z] [2026-09-04T18:33:19.287Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-09-04T18:33:19.287Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:19.287Z] [2026-09-04T18:33:19.287Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:33:19.287Z] ] *** [2026-09-04T18:33:20.474Z] skipping: [an-a01n02] [2026-09-04T18:33:20.474Z] ok: [an-a01n01] [2026-09-04T18:33:20.474Z] [2026-09-04T18:33:20.474Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T18:33:20.474Z] --ci-test \ [2026-09-04T18:33:20.474Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T18:33:20.474Z] --os almalinux9 \ [2026-09-04T18:33:20.474Z] --cpu 2 --ram 4G \ [2026-09-04T18:33:20.474Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T18:33:20.474Z] --install-media {{ installisouuid }} \ [2026-09-04T18:33:20.474Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T18:33:20.474Z] ] *** [2026-09-04T18:33:20.474Z] skipping: [an-a01n01] [2026-09-04T18:33:20.474Z] skipping: [an-a01n02] [2026-09-04T18:33:20.474Z] [2026-09-04T18:33:20.474Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T18:33:20.474Z] skipping: [an-a01n01] [2026-09-04T18:33:20.474Z] skipping: [an-a01n02] [2026-09-04T18:33:20.474Z] [2026-09-04T18:33:20.474Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:33:20.474Z] skipping: [an-a01n01] [2026-09-04T18:33:20.474Z] skipping: [an-a01n02] [2026-09-04T18:33:20.474Z] [2026-09-04T18:33:20.474Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T18:33:20.816Z] skipping: [an-a01n01] [2026-09-04T18:33:20.816Z] skipping: [an-a01n02] [2026-09-04T18:33:20.816Z] [2026-09-04T18:33:20.816Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T18:33:20.816Z] skipping: [an-a01n01] [2026-09-04T18:33:20.816Z] skipping: [an-a01n02] [2026-09-04T18:33:20.816Z] [2026-09-04T18:33:20.816Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-09-04T18:33:20.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:20.816Z] [2026-09-04T18:33:20.816Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:33:20.816Z] ] *** [2026-09-04T18:33:22.526Z] skipping: [an-a01n01] [2026-09-04T18:33:22.526Z] ok: [an-a01n02] [2026-09-04T18:33:22.526Z] [2026-09-04T18:33:22.526Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T18:33:22.526Z] --ci-test \ [2026-09-04T18:33:22.526Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T18:33:22.526Z] --os almalinux9 \ [2026-09-04T18:33:22.526Z] --cpu 2 --ram 4G \ [2026-09-04T18:33:22.526Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T18:33:22.526Z] --install-media {{ installisouuid }} \ [2026-09-04T18:33:22.526Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T18:33:22.526Z] ] *** [2026-09-04T18:33:22.526Z] skipping: [an-a01n01] [2026-09-04T18:33:22.526Z] skipping: [an-a01n02] [2026-09-04T18:33:22.526Z] [2026-09-04T18:33:22.526Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T18:33:22.526Z] skipping: [an-a01n01] [2026-09-04T18:33:22.526Z] skipping: [an-a01n02] [2026-09-04T18:33:22.526Z] [2026-09-04T18:33:22.526Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:33:22.526Z] skipping: [an-a01n01] [2026-09-04T18:33:22.526Z] skipping: [an-a01n02] [2026-09-04T18:33:22.526Z] [2026-09-04T18:33:22.526Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T18:33:22.868Z] skipping: [an-a01n01] [2026-09-04T18:33:22.868Z] skipping: [an-a01n02] [2026-09-04T18:33:22.868Z] [2026-09-04T18:33:22.868Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T18:33:22.868Z] skipping: [an-a01n01] [2026-09-04T18:33:22.868Z] skipping: [an-a01n02] [2026-09-04T18:33:22.868Z] [2026-09-04T18:33:22.868Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-09-04T18:33:22.868Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:22.868Z] [2026-09-04T18:33:22.868Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:33:22.868Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:33:22.868Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:33:22.868Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:33:22.868Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:33:22.868Z] echo "pcs:" [2026-09-04T18:33:22.868Z] echo "$pcsstatus" [2026-09-04T18:33:22.868Z] exit 1 [2026-09-04T18:33:22.868Z] fi [2026-09-04T18:33:22.868Z] exit 0 [2026-09-04T18:33:22.868Z] ] *** [2026-09-04T18:33:25.188Z] ok: [an-a01n02] [2026-09-04T18:33:25.188Z] ok: [an-a01n01] [2026-09-04T18:33:25.188Z] [2026-09-04T18:33:25.188Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:33:25.188Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:33:25.188Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:33:25.188Z] ret=0 [2026-09-04T18:33:25.188Z] if [ "$servers" != "1" ]; then [2026-09-04T18:33:25.188Z] echo "Not all servers are running on the same node!" [2026-09-04T18:33:25.188Z] echo "$pcsstatus" [2026-09-04T18:33:25.188Z] ret=1 [2026-09-04T18:33:25.188Z] fi [2026-09-04T18:33:25.188Z] echo "$hypervisor" [2026-09-04T18:33:25.188Z] exit $ret [2026-09-04T18:33:25.188Z] ] *** [2026-09-04T18:33:27.509Z] ok: [an-a01n01] [2026-09-04T18:33:27.509Z] ok: [an-a01n02] [2026-09-04T18:33:27.509Z] [2026-09-04T18:33:27.509Z] PLAY [delete extra server tests] *********************************************** [2026-09-04T18:33:27.509Z] [2026-09-04T18:33:27.509Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:33:29.220Z] ok: [an-a01n01] [2026-09-04T18:33:29.220Z] ok: [an-a01n02] [2026-09-04T18:33:29.220Z] [2026-09-04T18:33:29.220Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T18:33:29.220Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:33:29.220Z] [2026-09-04T18:33:29.220Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:33:29.220Z] ] *** [2026-09-04T18:33:34.078Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:33:34.078Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:33:34.078Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:33:34.078Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:33:34.078Z] [2026-09-04T18:33:34.078Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-09-04T18:33:34.078Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T18:33:34.078Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T18:33:34.078Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T18:33:34.078Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T18:33:34.078Z] [2026-09-04T18:33:34.078Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:33:34.078Z] ] *** [2026-09-04T18:33:35.265Z] skipping: [an-a01n01] [2026-09-04T18:33:35.265Z] ok: [an-a01n02] [2026-09-04T18:33:35.265Z] [2026-09-04T18:33:35.265Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T18:33:35.265Z] --force \ [2026-09-04T18:33:35.265Z] --server an-test-deploy{{ servernum }} [2026-09-04T18:33:35.265Z] ] *** [2026-09-04T18:33:44.388Z] skipping: [an-a01n01] [2026-09-04T18:33:44.388Z] changed: [an-a01n02] [2026-09-04T18:33:44.388Z] [2026-09-04T18:33:44.388Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:33:44.388Z] ] *** [2026-09-04T18:34:12.382Z] skipping: [an-a01n01] [2026-09-04T18:34:12.382Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-09-04T18:34:12.382Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-09-04T18:34:12.382Z] ok: [an-a01n02] [2026-09-04T18:34:12.382Z] [2026-09-04T18:34:12.382Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:34:12.382Z] ] *** [2026-09-04T18:34:12.382Z] skipping: [an-a01n01] [2026-09-04T18:34:12.382Z] ok: [an-a01n02] [2026-09-04T18:34:12.382Z] [2026-09-04T18:34:12.382Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:34:12.382Z] ] *** [2026-09-04T18:34:12.382Z] skipping: [an-a01n01] [2026-09-04T18:34:12.382Z] ok: [an-a01n02] [2026-09-04T18:34:12.382Z] [2026-09-04T18:34:12.382Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:34:12.382Z] ] *** [2026-09-04T18:34:12.382Z] skipping: [an-a01n01] [2026-09-04T18:34:12.382Z] ok: [an-a01n02] [2026-09-04T18:34:12.382Z] [2026-09-04T18:34:12.382Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T18:34:12.382Z] --force \ [2026-09-04T18:34:12.382Z] --server an-test-deploy{{ servernum }} [2026-09-04T18:34:12.382Z] ] *** [2026-09-04T18:34:21.418Z] skipping: [an-a01n01] [2026-09-04T18:34:21.418Z] changed: [an-a01n02] [2026-09-04T18:34:21.418Z] [2026-09-04T18:34:21.418Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:34:21.418Z] ] *** [2026-09-04T18:35:44.798Z] skipping: [an-a01n01] [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-09-04T18:35:44.799Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-09-04T18:35:44.799Z] ok: [an-a01n02] [2026-09-04T18:35:44.799Z] [2026-09-04T18:35:44.799Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:35:44.799Z] ] *** [2026-09-04T18:35:44.799Z] skipping: [an-a01n01] [2026-09-04T18:35:44.799Z] ok: [an-a01n02] [2026-09-04T18:35:44.799Z] [2026-09-04T18:35:44.799Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:35:44.799Z] ] *** [2026-09-04T18:35:44.799Z] skipping: [an-a01n01] [2026-09-04T18:35:44.799Z] ok: [an-a01n02] [2026-09-04T18:35:44.799Z] [2026-09-04T18:35:44.799Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:35:44.799Z] ] *** [2026-09-04T18:35:44.799Z] skipping: [an-a01n01] [2026-09-04T18:35:44.799Z] ok: [an-a01n02] [2026-09-04T18:35:44.799Z] [2026-09-04T18:35:44.799Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T18:35:44.799Z] --force \ [2026-09-04T18:35:44.799Z] --server an-test-deploy{{ servernum }} [2026-09-04T18:35:44.799Z] ] *** [2026-09-04T18:35:44.799Z] skipping: [an-a01n01] [2026-09-04T18:35:44.799Z] changed: [an-a01n02] [2026-09-04T18:35:44.799Z] [2026-09-04T18:35:44.799Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:35:44.799Z] ] *** [2026-09-04T18:37:08.245Z] skipping: [an-a01n01] [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-09-04T18:37:08.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (55 retries left). [2026-09-04T18:37:08.245Z] ok: [an-a01n02] [2026-09-04T18:37:08.245Z] [2026-09-04T18:37:08.245Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:37:08.245Z] ] *** [2026-09-04T18:37:08.245Z] skipping: [an-a01n01] [2026-09-04T18:37:08.245Z] ok: [an-a01n02] [2026-09-04T18:37:08.245Z] [2026-09-04T18:37:08.245Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:37:08.245Z] ] *** [2026-09-04T18:37:08.245Z] skipping: [an-a01n01] [2026-09-04T18:37:08.245Z] ok: [an-a01n02] [2026-09-04T18:37:08.245Z] [2026-09-04T18:37:08.245Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:37:08.245Z] ] *** [2026-09-04T18:37:08.245Z] skipping: [an-a01n01] [2026-09-04T18:37:08.245Z] ok: [an-a01n02] [2026-09-04T18:37:08.245Z] [2026-09-04T18:37:08.245Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T18:37:08.245Z] --force \ [2026-09-04T18:37:08.245Z] --server an-test-deploy{{ servernum }} [2026-09-04T18:37:08.245Z] ] *** [2026-09-04T18:37:08.245Z] skipping: [an-a01n01] [2026-09-04T18:37:08.245Z] changed: [an-a01n02] [2026-09-04T18:37:08.245Z] [2026-09-04T18:37:08.245Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:37:08.245Z] ] *** [2026-09-04T18:38:06.226Z] skipping: [an-a01n01] [2026-09-04T18:38:06.226Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-09-04T18:38:06.226Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-09-04T18:38:06.226Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-09-04T18:38:06.226Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-09-04T18:38:06.226Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-09-04T18:38:06.226Z] ok: [an-a01n02] [2026-09-04T18:38:06.226Z] [2026-09-04T18:38:06.226Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:38:06.226Z] ] *** [2026-09-04T18:38:06.568Z] skipping: [an-a01n01] [2026-09-04T18:38:06.568Z] ok: [an-a01n02] [2026-09-04T18:38:06.568Z] [2026-09-04T18:38:06.568Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:38:06.568Z] ] *** [2026-09-04T18:38:19.950Z] skipping: [an-a01n01] [2026-09-04T18:38:19.950Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed (60 retries left). [2026-09-04T18:38:19.950Z] ok: [an-a01n02] [2026-09-04T18:38:19.950Z] [2026-09-04T18:38:19.950Z] PLAY [create server for DR testing] ******************************************** [2026-09-04T18:38:19.950Z] [2026-09-04T18:38:19.950Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:38:20.292Z] ok: [an-a01n02] [2026-09-04T18:38:20.292Z] ok: [an-a01n01] [2026-09-04T18:38:20.292Z] [2026-09-04T18:38:20.292Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T18:38:20.634Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:38:20.634Z] [2026-09-04T18:38:20.634Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:38:20.634Z] ] *** [2026-09-04T18:38:24.504Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:38:24.504Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:38:24.504Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:38:24.504Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:38:24.504Z] [2026-09-04T18:38:24.504Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-09-04T18:38:24.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-09-04T18:38:24.504Z] [2026-09-04T18:38:24.504Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-09-04T18:38:24.504Z] if [ -z "$storage" ]; then [2026-09-04T18:38:24.504Z] echo "Unable to detect shared storage" [2026-09-04T18:38:24.504Z] exit 1 [2026-09-04T18:38:24.504Z] fi [2026-09-04T18:38:24.504Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:38:24.504Z] ] *** [2026-09-04T18:38:30.524Z] changed: [an-a01n02] [2026-09-04T18:38:30.524Z] changed: [an-a01n01] [2026-09-04T18:38:30.524Z] [2026-09-04T18:38:30.524Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-09-04T18:38:30.524Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-09-04T18:38:30.524Z] [2026-09-04T18:38:30.524Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-09-04T18:38:30.524Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-09-04T18:38:30.524Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-09-04T18:38:30.524Z] echo "Unable to detect all deploy iso" [2026-09-04T18:38:30.524Z] echo "Detected isos" [2026-09-04T18:38:30.524Z] echo "$isos" [2026-09-04T18:38:30.524Z] exit 1 [2026-09-04T18:38:30.524Z] fi [2026-09-04T18:38:30.524Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-09-04T18:38:30.524Z] if [ -z "$installiso" ]; then [2026-09-04T18:38:30.524Z] echo "Unable to detect install iso" [2026-09-04T18:38:30.524Z] echo "Detected isos" [2026-09-04T18:38:30.524Z] echo "$isos" [2026-09-04T18:38:30.524Z] exit 1 [2026-09-04T18:38:30.524Z] fi [2026-09-04T18:38:30.524Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:38:30.524Z] ] *** [2026-09-04T18:38:36.545Z] changed: [an-a01n01] [2026-09-04T18:38:36.545Z] changed: [an-a01n02] [2026-09-04T18:38:36.545Z] [2026-09-04T18:38:36.545Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-09-04T18:38:36.545Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-09-04T18:38:36.545Z] [2026-09-04T18:38:36.545Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:38:36.545Z] ] *** [2026-09-04T18:38:37.278Z] skipping: [an-a01n02] [2026-09-04T18:38:37.278Z] ok: [an-a01n01] [2026-09-04T18:38:37.278Z] [2026-09-04T18:38:37.278Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T18:38:37.278Z] --ci-test \ [2026-09-04T18:38:37.278Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T18:38:37.278Z] --os almalinux9 \ [2026-09-04T18:38:37.278Z] --cpu 2 --ram 4G \ [2026-09-04T18:38:37.278Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T18:38:37.278Z] --install-media {{ installisouuid }} \ [2026-09-04T18:38:37.278Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T18:38:37.278Z] ] *** [2026-09-04T18:38:37.622Z] skipping: [an-a01n01] [2026-09-04T18:38:37.622Z] skipping: [an-a01n02] [2026-09-04T18:38:37.622Z] [2026-09-04T18:38:37.622Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T18:38:37.622Z] skipping: [an-a01n01] [2026-09-04T18:38:37.622Z] skipping: [an-a01n02] [2026-09-04T18:38:37.622Z] [2026-09-04T18:38:37.622Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:38:37.622Z] skipping: [an-a01n01] [2026-09-04T18:38:37.622Z] skipping: [an-a01n02] [2026-09-04T18:38:37.622Z] [2026-09-04T18:38:37.622Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T18:38:37.622Z] skipping: [an-a01n01] [2026-09-04T18:38:37.622Z] skipping: [an-a01n02] [2026-09-04T18:38:37.622Z] [2026-09-04T18:38:37.622Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T18:38:37.622Z] skipping: [an-a01n01] [2026-09-04T18:38:37.622Z] skipping: [an-a01n02] [2026-09-04T18:38:37.622Z] [2026-09-04T18:38:37.622Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-09-04T18:38:37.964Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:38:37.964Z] [2026-09-04T18:38:37.964Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:38:37.964Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:38:37.964Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:38:37.964Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:38:37.964Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:38:37.964Z] echo "pcs:" [2026-09-04T18:38:37.964Z] echo "$pcsstatus" [2026-09-04T18:38:37.964Z] exit 1 [2026-09-04T18:38:37.964Z] fi [2026-09-04T18:38:37.964Z] exit 0 [2026-09-04T18:38:37.964Z] ] *** [2026-09-04T18:38:39.672Z] ok: [an-a01n02] [2026-09-04T18:38:39.672Z] ok: [an-a01n01] [2026-09-04T18:38:39.672Z] [2026-09-04T18:38:39.672Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:38:39.672Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:38:39.672Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:38:39.672Z] ret=0 [2026-09-04T18:38:39.672Z] if [ "$servers" != "1" ]; then [2026-09-04T18:38:39.672Z] echo "Not all servers are running on the same node!" [2026-09-04T18:38:39.672Z] echo "$pcsstatus" [2026-09-04T18:38:39.672Z] ret=1 [2026-09-04T18:38:39.672Z] fi [2026-09-04T18:38:39.672Z] echo "$hypervisor" [2026-09-04T18:38:39.672Z] exit $ret [2026-09-04T18:38:39.672Z] ] *** [2026-09-04T18:38:41.993Z] ok: [an-a01n02] [2026-09-04T18:38:41.993Z] ok: [an-a01n01] [2026-09-04T18:38:41.993Z] [2026-09-04T18:38:41.993Z] PLAY [DR tests] **************************************************************** [2026-09-04T18:38:41.993Z] [2026-09-04T18:38:41.993Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:38:44.313Z] ok: [an-a01dr01] [2026-09-04T18:38:44.313Z] [2026-09-04T18:38:44.313Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured [2026-09-04T18:38:44.313Z] ] *** [2026-09-04T18:38:47.346Z] changed: [an-a01dr01] [2026-09-04T18:38:47.346Z] [2026-09-04T18:38:47.346Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g') [2026-09-04T18:38:47.346Z] echo "DB registered hostname: $drhostname" [2026-09-04T18:38:47.346Z] echo "current hostname: $(hostname)" [2026-09-04T18:38:47.346Z] if [ "$drhostname" != "$(hostname)" ]; then [2026-09-04T18:38:47.346Z] echo "DR hostnames are not matching" [2026-09-04T18:38:47.346Z] exit 1 [2026-09-04T18:38:47.346Z] fi [2026-09-04T18:38:47.346Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then [2026-09-04T18:38:47.346Z] echo "DR hostnames are not matching drnodes an-a01dr01" [2026-09-04T18:38:47.346Z] exit 1 [2026-09-04T18:38:47.346Z] fi [2026-09-04T18:38:47.346Z] ] *** [2026-09-04T18:38:52.226Z] changed: [an-a01dr01] [2026-09-04T18:38:52.226Z] [2026-09-04T18:38:52.226Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-09-04T18:38:52.226Z] echo "Detected a linked DR host!" [2026-09-04T18:38:52.226Z] exit 1 [2026-09-04T18:38:52.226Z] fi [2026-09-04T18:38:52.226Z] ] *** [2026-09-04T18:38:56.097Z] changed: [an-a01dr01] [2026-09-04T18:38:56.097Z] [2026-09-04T18:38:56.097Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01 [2026-09-04T18:38:56.097Z] ] *** [2026-09-04T18:39:00.952Z] changed: [an-a01dr01] [2026-09-04T18:39:00.952Z] [2026-09-04T18:39:00.952Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then [2026-09-04T18:39:00.952Z] echo "DR host failed to link to anvil" [2026-09-04T18:39:00.952Z] exit 1 [2026-09-04T18:39:00.952Z] fi [2026-09-04T18:39:00.952Z] ] *** [2026-09-04T18:39:05.804Z] changed: [an-a01dr01] [2026-09-04T18:39:05.804Z] [2026-09-04T18:39:05.804Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-09-04T18:39:05.804Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-09-04T18:39:05.804Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-09-04T18:39:05.804Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-09-04T18:39:05.804Z] exit 1 [2026-09-04T18:39:05.804Z] fi [2026-09-04T18:39:05.804Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then [2026-09-04T18:39:05.804Z] echo "Unable to link local vg uuid to storage group" [2026-09-04T18:39:05.804Z] exit 1 [2026-09-04T18:39:05.804Z] fi [2026-09-04T18:39:05.804Z] ] *** [2026-09-04T18:39:13.209Z] changed: [an-a01dr01] [2026-09-04T18:39:13.209Z] [2026-09-04T18:39:13.209Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then [2026-09-04T18:39:13.209Z] echo "Local vg does not appear to be linked to storage group" [2026-09-04T18:39:13.209Z] exit 1 [2026-09-04T18:39:13.209Z] fi [2026-09-04T18:39:13.209Z] ] *** [2026-09-04T18:39:16.240Z] changed: [an-a01dr01] [2026-09-04T18:39:16.240Z] [2026-09-04T18:39:16.240Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then [2026-09-04T18:39:16.240Z] echo "Unable to add protection to server an-test-deploy1" [2026-09-04T18:39:16.240Z] exit 1 [2026-09-04T18:39:16.240Z] fi [2026-09-04T18:39:16.240Z] ] *** [2026-09-04T18:39:21.185Z] changed: [an-a01dr01] [2026-09-04T18:39:21.185Z] [2026-09-04T18:39:21.185Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-09-04T18:39:21.185Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-09-04T18:39:21.185Z] [2026-09-04T18:39:21.185Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-09-04T18:39:21.185Z] exit 1 [2026-09-04T18:39:21.185Z] fi [2026-09-04T18:39:21.185Z] ] *** [2026-09-04T18:39:54.761Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-09-04T18:39:54.761Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-09-04T18:39:54.761Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-09-04T18:39:54.761Z] ok: [an-a01dr01] [2026-09-04T18:39:54.761Z] [2026-09-04T18:39:54.761Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-09-04T18:39:54.761Z] short-throw) [2026-09-04T18:39:54.761Z] expected=2 [2026-09-04T18:39:54.761Z] protocol=A [2026-09-04T18:39:54.761Z] ;; [2026-09-04T18:39:54.761Z] sync) [2026-09-04T18:39:54.761Z] expected=3 [2026-09-04T18:39:54.761Z] protocol=C [2026-09-04T18:39:54.761Z] ;; [2026-09-04T18:39:54.761Z] *) [2026-09-04T18:39:54.761Z] echo "unknown DRBD protection method" [2026-09-04T18:39:54.761Z] exit 1 [2026-09-04T18:39:54.761Z] ;; [2026-09-04T18:39:54.761Z] esac [2026-09-04T18:39:54.761Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-09-04T18:39:54.761Z] echo "Unable to confirm correct DRBD protocol" [2026-09-04T18:39:54.761Z] exit 1 [2026-09-04T18:39:54.761Z] fi [2026-09-04T18:39:54.761Z] ] *** [2026-09-04T18:39:54.761Z] changed: [an-a01dr01] [2026-09-04T18:39:54.761Z] [2026-09-04T18:39:54.761Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:39:54.761Z] ] *** [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (107 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (106 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (105 retries left). [2026-09-04T18:43:03.860Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (104 retries left). [2026-09-04T18:43:03.860Z] ok: [an-a01dr01] [2026-09-04T18:43:03.860Z] [2026-09-04T18:43:03.860Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-09-04T18:43:03.860Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-09-04T18:43:03.860Z] [2026-09-04T18:43:03.860Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:43:03.860Z] ] *** [2026-09-04T18:43:03.860Z] ok: [an-a01dr01] [2026-09-04T18:43:03.860Z] [2026-09-04T18:43:03.860Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-09-04T18:43:03.860Z] echo "Unable to disconnect server from DR replication" [2026-09-04T18:43:03.860Z] exit 1 [2026-09-04T18:43:03.860Z] fi [2026-09-04T18:43:03.860Z] ] *** [2026-09-04T18:43:03.860Z] changed: [an-a01dr01] [2026-09-04T18:43:03.860Z] [2026-09-04T18:43:03.860Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:43:03.860Z] ] *** [2026-09-04T18:43:37.458Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-09-04T18:43:37.458Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-09-04T18:43:37.458Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (118 retries left). [2026-09-04T18:43:37.458Z] ok: [an-a01dr01] [2026-09-04T18:43:37.458Z] [2026-09-04T18:43:37.458Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-09-04T18:43:37.458Z] sleep 120 [2026-09-04T18:43:37.458Z] ] *** [2026-09-04T18:45:46.123Z] changed: [an-a01dr01 -> localhost] [2026-09-04T18:45:46.123Z] [2026-09-04T18:45:46.123Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-09-04T18:45:46.123Z] echo "Unable to update server DR replications" [2026-09-04T18:45:46.123Z] exit 1 [2026-09-04T18:45:46.123Z] fi [2026-09-04T18:45:46.123Z] ] *** [2026-09-04T18:45:48.443Z] changed: [an-a01dr01] [2026-09-04T18:45:48.443Z] [2026-09-04T18:45:48.443Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:45:48.443Z] ] *** [2026-09-04T18:46:11.807Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-09-04T18:46:11.807Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-09-04T18:46:11.807Z] ok: [an-a01dr01] [2026-09-04T18:46:11.807Z] [2026-09-04T18:46:11.807Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:46:11.807Z] ] *** [2026-09-04T18:46:22.782Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-09-04T18:46:22.782Z] ok: [an-a01dr01] [2026-09-04T18:46:22.782Z] [2026-09-04T18:46:22.782Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-09-04T18:46:22.782Z] echo "Unable to (re)connect server from DR replication" [2026-09-04T18:46:22.782Z] exit 1 [2026-09-04T18:46:22.782Z] fi [2026-09-04T18:46:22.782Z] ] *** [2026-09-04T18:46:27.634Z] changed: [an-a01dr01] [2026-09-04T18:46:27.634Z] [2026-09-04T18:46:27.634Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:46:27.634Z] ] *** [2026-09-04T18:46:50.983Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-09-04T18:46:50.983Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-09-04T18:46:50.983Z] ok: [an-a01dr01] [2026-09-04T18:46:50.983Z] [2026-09-04T18:46:50.983Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-09-04T18:46:50.983Z] echo "Unable to remove protection from server an-test-deploy1" [2026-09-04T18:46:50.983Z] exit 1 [2026-09-04T18:46:50.983Z] fi [2026-09-04T18:46:50.983Z] ] *** [2026-09-04T18:46:55.836Z] changed: [an-a01dr01] [2026-09-04T18:46:55.837Z] [2026-09-04T18:46:55.837Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-09-04T18:46:55.837Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-09-04T18:46:55.837Z] [2026-09-04T18:46:55.837Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:46:55.837Z] ] *** [2026-09-04T18:47:54.553Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-09-04T18:47:54.553Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-09-04T18:47:54.553Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-09-04T18:47:54.553Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (57 retries left). [2026-09-04T18:47:54.553Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (56 retries left). [2026-09-04T18:47:54.553Z] ok: [an-a01dr01] [2026-09-04T18:47:54.553Z] [2026-09-04T18:47:54.553Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:47:54.553Z] ] *** [2026-09-04T18:47:54.553Z] ok: [an-a01dr01] [2026-09-04T18:47:54.553Z] [2026-09-04T18:47:54.553Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-09-04T18:47:54.553Z] exit 1 [2026-09-04T18:47:54.553Z] fi [2026-09-04T18:47:54.553Z] ] *** [2026-09-04T18:48:22.508Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-09-04T18:48:22.508Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-09-04T18:48:22.508Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-09-04T18:48:22.508Z] ok: [an-a01dr01] [2026-09-04T18:48:22.508Z] [2026-09-04T18:48:22.508Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-09-04T18:48:22.508Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-09-04T18:48:22.508Z] #exit 1 [2026-09-04T18:48:22.508Z] fi [2026-09-04T18:48:22.508Z] ] *** [2026-09-04T18:48:22.850Z] changed: [an-a01dr01 -> an-a01n01] [2026-09-04T18:48:22.850Z] [2026-09-04T18:48:22.850Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then [2026-09-04T18:48:22.850Z] echo "Unable to add protection to server an-test-deploy1" [2026-09-04T18:48:22.850Z] exit 1 [2026-09-04T18:48:22.850Z] fi [2026-09-04T18:48:22.850Z] ] *** [2026-09-04T18:48:28.870Z] changed: [an-a01dr01] [2026-09-04T18:48:28.870Z] [2026-09-04T18:48:28.870Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-09-04T18:48:28.870Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-09-04T18:48:28.870Z] [2026-09-04T18:48:28.870Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-09-04T18:48:28.870Z] exit 1 [2026-09-04T18:48:28.870Z] fi [2026-09-04T18:48:28.870Z] ] *** [2026-09-04T18:49:02.484Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-09-04T18:49:02.484Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-09-04T18:49:02.484Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-09-04T18:49:02.484Z] ok: [an-a01dr01] [2026-09-04T18:49:02.484Z] [2026-09-04T18:49:02.484Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-09-04T18:49:02.484Z] short-throw) [2026-09-04T18:49:02.484Z] expected=2 [2026-09-04T18:49:02.484Z] protocol=A [2026-09-04T18:49:02.484Z] ;; [2026-09-04T18:49:02.484Z] sync) [2026-09-04T18:49:02.484Z] expected=3 [2026-09-04T18:49:02.484Z] protocol=C [2026-09-04T18:49:02.484Z] ;; [2026-09-04T18:49:02.484Z] *) [2026-09-04T18:49:02.484Z] echo "unknown DRBD protection method" [2026-09-04T18:49:02.484Z] exit 1 [2026-09-04T18:49:02.484Z] ;; [2026-09-04T18:49:02.484Z] esac [2026-09-04T18:49:02.484Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-09-04T18:49:02.484Z] echo "Unable to confirm correct DRBD protocol" [2026-09-04T18:49:02.484Z] exit 1 [2026-09-04T18:49:02.484Z] fi [2026-09-04T18:49:02.484Z] ] *** [2026-09-04T18:49:02.484Z] changed: [an-a01dr01] [2026-09-04T18:49:02.484Z] [2026-09-04T18:49:02.484Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:49:02.484Z] ] *** [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-09-04T18:51:11.143Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-09-04T18:51:11.143Z] ok: [an-a01dr01] [2026-09-04T18:51:11.143Z] [2026-09-04T18:51:11.143Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-09-04T18:51:11.143Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-09-04T18:51:11.143Z] [2026-09-04T18:51:11.143Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T18:51:11.143Z] ] *** [2026-09-04T18:51:11.143Z] ok: [an-a01dr01] [2026-09-04T18:51:11.143Z] [2026-09-04T18:51:11.143Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-09-04T18:51:11.143Z] echo "Unable to disconnect server from DR replication" [2026-09-04T18:51:11.143Z] exit 1 [2026-09-04T18:51:11.143Z] fi [2026-09-04T18:51:11.143Z] ] *** [2026-09-04T18:51:18.538Z] changed: [an-a01dr01] [2026-09-04T18:51:18.538Z] [2026-09-04T18:51:18.538Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:51:18.538Z] ] *** [2026-09-04T18:51:52.285Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-09-04T18:51:52.285Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-09-04T18:51:52.285Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (118 retries left). [2026-09-04T18:51:52.285Z] ok: [an-a01dr01] [2026-09-04T18:51:52.285Z] [2026-09-04T18:51:52.285Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-09-04T18:51:52.285Z] sleep 120 [2026-09-04T18:51:52.285Z] ] *** [2026-09-04T18:54:00.919Z] changed: [an-a01dr01 -> localhost] [2026-09-04T18:54:00.919Z] [2026-09-04T18:54:00.919Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-09-04T18:54:00.919Z] echo "Unable to update server DR replications" [2026-09-04T18:54:00.919Z] exit 1 [2026-09-04T18:54:00.919Z] fi [2026-09-04T18:54:00.919Z] ] *** [2026-09-04T18:54:02.127Z] changed: [an-a01dr01] [2026-09-04T18:54:02.127Z] [2026-09-04T18:54:02.127Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:54:02.127Z] ] *** [2026-09-04T18:54:25.446Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-09-04T18:54:25.446Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-09-04T18:54:25.446Z] ok: [an-a01dr01] [2026-09-04T18:54:25.446Z] [2026-09-04T18:54:25.446Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:54:25.446Z] ] *** [2026-09-04T18:54:48.791Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-09-04T18:54:48.791Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-09-04T18:54:48.791Z] ok: [an-a01dr01] [2026-09-04T18:54:48.791Z] [2026-09-04T18:54:48.791Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-09-04T18:54:48.791Z] echo "Unable to (re)connect server from DR replication" [2026-09-04T18:54:48.791Z] exit 1 [2026-09-04T18:54:48.791Z] fi [2026-09-04T18:54:48.791Z] ] *** [2026-09-04T18:54:51.114Z] changed: [an-a01dr01] [2026-09-04T18:54:51.114Z] [2026-09-04T18:54:51.114Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:54:51.114Z] ] *** [2026-09-04T18:55:15.066Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-09-04T18:55:15.066Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-09-04T18:55:15.066Z] ok: [an-a01dr01] [2026-09-04T18:55:15.066Z] [2026-09-04T18:55:15.066Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-09-04T18:55:15.066Z] echo "Unable to remove protection from server an-test-deploy1" [2026-09-04T18:55:15.066Z] exit 1 [2026-09-04T18:55:15.066Z] fi [2026-09-04T18:55:15.066Z] ] *** [2026-09-04T18:55:18.936Z] changed: [an-a01dr01] [2026-09-04T18:55:18.936Z] [2026-09-04T18:55:18.936Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-09-04T18:55:18.936Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-09-04T18:55:18.936Z] [2026-09-04T18:55:18.936Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T18:55:18.936Z] ] *** [2026-09-04T18:55:52.865Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-09-04T18:55:52.865Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-09-04T18:55:52.865Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-09-04T18:55:52.865Z] ok: [an-a01dr01] [2026-09-04T18:55:52.865Z] [2026-09-04T18:55:52.865Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T18:55:52.865Z] ] *** [2026-09-04T18:56:03.839Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed (60 retries left). [2026-09-04T18:56:03.839Z] ok: [an-a01dr01] [2026-09-04T18:56:03.839Z] [2026-09-04T18:56:03.840Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-09-04T18:56:03.840Z] exit 1 [2026-09-04T18:56:03.840Z] fi [2026-09-04T18:56:03.840Z] ] *** [2026-09-04T18:56:37.417Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-09-04T18:56:37.417Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-09-04T18:56:37.417Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-09-04T18:56:37.417Z] ok: [an-a01dr01] [2026-09-04T18:56:37.417Z] [2026-09-04T18:56:37.417Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-09-04T18:56:37.417Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-09-04T18:56:37.417Z] #exit 1 [2026-09-04T18:56:37.417Z] fi [2026-09-04T18:56:37.417Z] ] *** [2026-09-04T18:56:37.417Z] changed: [an-a01dr01 -> an-a01n01] [2026-09-04T18:56:37.417Z] [2026-09-04T18:56:37.417Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-09-04T18:56:37.417Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-09-04T18:56:37.417Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-09-04T18:56:37.417Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-09-04T18:56:37.417Z] exit 1 [2026-09-04T18:56:37.417Z] fi [2026-09-04T18:56:37.417Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then [2026-09-04T18:56:37.417Z] echo "Unable to unlink local vg uuid to storage group" [2026-09-04T18:56:37.417Z] exit 1 [2026-09-04T18:56:37.417Z] fi [2026-09-04T18:56:37.417Z] ] *** [2026-09-04T18:56:43.440Z] changed: [an-a01dr01] [2026-09-04T18:56:43.440Z] [2026-09-04T18:56:43.440Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then [2026-09-04T18:56:43.440Z] echo "Local vg does not appear to be unlinked from storage group (#886)" [2026-09-04T18:56:43.440Z] exit 1 [2026-09-04T18:56:43.440Z] fi [2026-09-04T18:56:43.440Z] ] *** [2026-09-04T18:56:46.473Z] changed: [an-a01dr01] [2026-09-04T18:56:46.473Z] [2026-09-04T18:56:46.473Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01 [2026-09-04T18:56:46.473Z] ] *** [2026-09-04T18:56:52.489Z] changed: [an-a01dr01] [2026-09-04T18:56:52.489Z] [2026-09-04T18:56:52.489Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-09-04T18:56:52.489Z] echo "Detected a linked DR host!" [2026-09-04T18:56:52.489Z] exit 1 [2026-09-04T18:56:52.489Z] fi [2026-09-04T18:56:52.489Z] ] *** [2026-09-04T18:56:56.360Z] changed: [an-a01dr01] [2026-09-04T18:56:56.360Z] [2026-09-04T18:56:56.360Z] PLAY [upgrade server test] ***************************************************** [2026-09-04T18:56:56.360Z] [2026-09-04T18:56:56.360Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:56:58.679Z] ok: [an-a01n01] [2026-09-04T18:56:58.679Z] ok: [an-a01n02] [2026-09-04T18:56:58.679Z] [2026-09-04T18:56:58.679Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T18:56:58.679Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T18:56:58.679Z] [2026-09-04T18:56:58.679Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T18:56:58.679Z] ] *** [2026-09-04T18:57:02.771Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T18:57:02.771Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T18:57:02.771Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T18:57:02.771Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T18:57:02.771Z] [2026-09-04T18:57:02.771Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-09-04T18:57:02.771Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-09-04T18:57:02.771Z] [2026-09-04T18:57:02.771Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-09-04T18:57:02.771Z] if [ -z "$storage" ]; then [2026-09-04T18:57:02.771Z] echo "Unable to detect shared storage" [2026-09-04T18:57:02.771Z] exit 1 [2026-09-04T18:57:02.771Z] fi [2026-09-04T18:57:02.771Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:57:02.771Z] ] *** [2026-09-04T18:57:07.630Z] changed: [an-a01n01] [2026-09-04T18:57:07.630Z] changed: [an-a01n02] [2026-09-04T18:57:07.630Z] [2026-09-04T18:57:07.630Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-09-04T18:57:07.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-09-04T18:57:07.630Z] [2026-09-04T18:57:07.630Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-09-04T18:57:07.630Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-09-04T18:57:07.630Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-09-04T18:57:07.630Z] echo "Unable to detect all deploy iso" [2026-09-04T18:57:07.630Z] echo "Detected isos" [2026-09-04T18:57:07.630Z] echo "$isos" [2026-09-04T18:57:07.630Z] exit 1 [2026-09-04T18:57:07.630Z] fi [2026-09-04T18:57:07.630Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-09-04T18:57:07.630Z] if [ -z "$installiso" ]; then [2026-09-04T18:57:07.630Z] echo "Unable to detect install iso" [2026-09-04T18:57:07.630Z] echo "Detected isos" [2026-09-04T18:57:07.630Z] echo "$isos" [2026-09-04T18:57:07.630Z] exit 1 [2026-09-04T18:57:07.630Z] fi [2026-09-04T18:57:07.630Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-09-04T18:57:07.630Z] ] *** [2026-09-04T18:57:13.651Z] changed: [an-a01n02] [2026-09-04T18:57:13.651Z] changed: [an-a01n01] [2026-09-04T18:57:13.651Z] [2026-09-04T18:57:13.651Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-09-04T18:57:13.651Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-09-04T18:57:13.651Z] [2026-09-04T18:57:13.651Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T18:57:13.651Z] ] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] ok: [an-a01n01] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 595557fc-2983-4003-a042-d555aa2378e9 _raw_params=anvil-provision-server -vv --log-secure \ [2026-09-04T18:57:15.369Z] --ci-test \ [2026-09-04T18:57:15.369Z] --name an-test-deploy{{ servernum }} \ [2026-09-04T18:57:15.369Z] --os almalinux9 \ [2026-09-04T18:57:15.369Z] --cpu 2 --ram 4G \ [2026-09-04T18:57:15.369Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-09-04T18:57:15.369Z] --install-media {{ installisouuid }} \ [2026-09-04T18:57:15.369Z] --driver-disc deploy{{ servernum }}.iso [2026-09-04T18:57:15.369Z] ] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n01] [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n01] [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n01] [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n01] [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-09-04T18:57:15.369Z] skipping: [an-a01n01] [2026-09-04T18:57:15.369Z] skipping: [an-a01n02] [2026-09-04T18:57:15.369Z] [2026-09-04T18:57:15.369Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-09-04T18:57:15.712Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-09-04T18:57:15.712Z] [2026-09-04T18:57:15.712Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:57:15.712Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-09-04T18:57:15.712Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-09-04T18:57:15.712Z] if [ "$numservers" != "$startedservers" ]; then [2026-09-04T18:57:15.712Z] echo "Detected $numservers but only $startedservers in Started state" [2026-09-04T18:57:15.712Z] echo "pcs:" [2026-09-04T18:57:15.712Z] echo "$pcsstatus" [2026-09-04T18:57:15.712Z] exit 1 [2026-09-04T18:57:15.712Z] fi [2026-09-04T18:57:15.712Z] exit 0 [2026-09-04T18:57:15.712Z] ] *** [2026-09-04T18:57:17.545Z] ok: [an-a01n02] [2026-09-04T18:57:17.545Z] ok: [an-a01n01] [2026-09-04T18:57:17.545Z] [2026-09-04T18:57:17.545Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-09-04T18:57:17.545Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-09-04T18:57:17.545Z] servers="$(echo "$hypervisor" | wc -l)" [2026-09-04T18:57:17.545Z] ret=0 [2026-09-04T18:57:17.545Z] if [ "$servers" != "1" ]; then [2026-09-04T18:57:17.545Z] echo "Not all servers are running on the same node!" [2026-09-04T18:57:17.545Z] echo "$pcsstatus" [2026-09-04T18:57:17.545Z] ret=1 [2026-09-04T18:57:17.545Z] fi [2026-09-04T18:57:17.545Z] echo "$hypervisor" [2026-09-04T18:57:17.545Z] exit $ret [2026-09-04T18:57:17.545Z] ] *** [2026-09-04T18:57:20.549Z] ok: [an-a01n01] [2026-09-04T18:57:20.549Z] ok: [an-a01n02] [2026-09-04T18:57:20.549Z] [2026-09-04T18:57:20.549Z] PLAY [enable upgrade repo] ***************************************************** [2026-09-04T18:57:20.549Z] [2026-09-04T18:57:20.549Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:57:22.260Z] ok: [an-a01dr01] [2026-09-04T18:57:22.260Z] ok: [an-a01n01] [2026-09-04T18:57:22.260Z] ok: [an-a01n02] [2026-09-04T18:57:22.260Z] ok: [an-striker01] [2026-09-04T18:57:22.260Z] ok: [an-striker02] [2026-09-04T18:57:22.260Z] [2026-09-04T18:57:22.260Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-09-04T18:57:22.260Z] dnf config-manager --set-enabled $updaterepo [2026-09-04T18:57:22.260Z] ] *** [2026-09-04T18:57:33.242Z] changed: [an-a01dr01] [2026-09-04T18:57:33.242Z] changed: [an-a01n02] [2026-09-04T18:57:33.242Z] changed: [an-a01n01] [2026-09-04T18:57:33.242Z] changed: [an-striker02] [2026-09-04T18:57:33.242Z] changed: [an-striker01] [2026-09-04T18:57:33.242Z] [2026-09-04T18:57:33.242Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-09-04T18:57:33.242Z] ok: [an-striker01] [2026-09-04T18:57:33.242Z] ok: [an-striker02] [2026-09-04T18:57:33.242Z] ok: [an-a01n01] [2026-09-04T18:57:33.242Z] ok: [an-a01n02] [2026-09-04T18:57:33.242Z] ok: [an-a01dr01] [2026-09-04T18:57:33.242Z] [2026-09-04T18:57:33.242Z] PLAY [start upgrade process] *************************************************** [2026-09-04T18:57:33.242Z] [2026-09-04T18:57:33.242Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T18:57:34.959Z] ok: [an-striker01] [2026-09-04T18:57:34.959Z] [2026-09-04T18:57:34.959Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-09-04T18:57:34.959Z] ] *** [2026-09-04T19:29:42.681Z] changed: [an-striker01] [2026-09-04T19:29:42.681Z] [2026-09-04T19:29:42.681Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-09-04T19:30:16.811Z] changed: [an-striker01] [2026-09-04T19:30:16.811Z] [2026-09-04T19:30:16.811Z] PLAY [reconnect all nodes] ***************************************************** [2026-09-04T19:30:16.811Z] [2026-09-04T19:30:16.811Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:30:19.131Z] ok: [an-a01dr01] [2026-09-04T19:30:19.131Z] ok: [an-a01n02] [2026-09-04T19:30:19.131Z] ok: [an-a01n01] [2026-09-04T19:30:19.131Z] ok: [an-striker02] [2026-09-04T19:30:19.131Z] ok: [an-striker01] [2026-09-04T19:30:19.131Z] [2026-09-04T19:30:19.131Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-09-04T19:30:19.131Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01 [2026-09-04T19:30:19.131Z] [2026-09-04T19:30:19.131Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-09-04T19:33:43.574Z] ok: [an-a01n02 -> localhost] [2026-09-04T19:33:43.574Z] ok: [an-striker01 -> localhost] [2026-09-04T19:33:43.574Z] ok: [an-striker02 -> localhost] [2026-09-04T19:33:43.574Z] ok: [an-a01dr01 -> localhost] [2026-09-04T19:33:43.574Z] ok: [an-a01n01 -> localhost] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-09-04T19:33:43.574Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-09-04T19:33:43.574Z] exit 1 [2026-09-04T19:33:43.574Z] fi [2026-09-04T19:33:43.574Z] ] *** [2026-09-04T19:33:43.574Z] changed: [an-a01n02] [2026-09-04T19:33:43.574Z] changed: [an-a01n01] [2026-09-04T19:33:43.574Z] changed: [an-a01dr01] [2026-09-04T19:33:43.574Z] changed: [an-striker01] [2026-09-04T19:33:43.574Z] changed: [an-striker02] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] PLAY [check all nodes status after upgrade] ************************************ [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:33:43.574Z] ok: [an-striker01] [2026-09-04T19:33:43.574Z] ok: [an-a01n02] [2026-09-04T19:33:43.574Z] ok: [an-a01dr01] [2026-09-04T19:33:43.574Z] ok: [an-a01n01] [2026-09-04T19:33:43.574Z] ok: [an-striker02] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-09-04T19:33:43.574Z] echo "perl-Log-Journald update not found!" [2026-09-04T19:33:43.574Z] exit 1 [2026-09-04T19:33:43.574Z] fi [2026-09-04T19:33:43.574Z] ] *** [2026-09-04T19:33:43.574Z] changed: [an-a01dr01] [2026-09-04T19:33:43.574Z] changed: [an-a01n02] [2026-09-04T19:33:43.574Z] changed: [an-a01n01] [2026-09-04T19:33:43.574Z] changed: [an-striker01] [2026-09-04T19:33:43.574Z] changed: [an-striker02] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] PLAY [check nodes status after upgrade] **************************************** [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:33:43.574Z] ok: [an-a01n02] [2026-09-04T19:33:43.574Z] ok: [an-a01n01] [2026-09-04T19:33:43.574Z] ok: [an-a01dr01] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-09-04T19:33:43.574Z] echo "akmod-drbd update not found!" [2026-09-04T19:33:43.574Z] exit 1 [2026-09-04T19:33:43.574Z] fi [2026-09-04T19:33:43.574Z] ] *** [2026-09-04T19:33:43.574Z] changed: [an-a01n02] [2026-09-04T19:33:43.574Z] changed: [an-a01dr01] [2026-09-04T19:33:43.574Z] changed: [an-a01n01] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] PLAY [check hosted node status after upgrade] ********************************** [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:33:43.574Z] ok: [an-a01n02] [2026-09-04T19:33:43.574Z] ok: [an-a01n01] [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T19:33:43.574Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T19:33:43.574Z] [2026-09-04T19:33:43.574Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T19:33:43.574Z] ] *** [2026-09-04T19:33:46.610Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T19:33:46.610Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T19:33:46.610Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T19:33:46.610Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T19:33:46.610Z] [2026-09-04T19:33:46.610Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-09-04T19:33:46.610Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-09-04T19:33:46.610Z] [2026-09-04T19:33:46.610Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-09-04T19:33:46.610Z] ] *** [2026-09-04T19:33:48.324Z] ok: [an-a01n02] [2026-09-04T19:33:48.324Z] ok: [an-a01n01] [2026-09-04T19:33:48.324Z] [2026-09-04T19:33:48.324Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-09-04T19:33:48.324Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-09-04T19:33:48.324Z] [2026-09-04T19:33:48.324Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-09-04T19:33:48.324Z] ] *** [2026-09-04T19:33:49.059Z] ok: [an-a01n02] [2026-09-04T19:33:49.059Z] ok: [an-a01n01] [2026-09-04T19:33:49.059Z] [2026-09-04T19:33:49.059Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-09-04T19:33:49.059Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-09-04T19:33:49.059Z] [2026-09-04T19:33:49.059Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-09-04T19:33:49.059Z] ] *** [2026-09-04T19:33:49.796Z] ok: [an-a01n02] [2026-09-04T19:33:49.796Z] ok: [an-a01n01] [2026-09-04T19:33:49.796Z] [2026-09-04T19:33:49.796Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-09-04T19:33:49.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-09-04T19:33:49.796Z] [2026-09-04T19:33:49.796Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-09-04T19:33:49.796Z] ] *** [2026-09-04T19:33:50.623Z] ok: [an-a01n01] [2026-09-04T19:33:50.623Z] ok: [an-a01n02] [2026-09-04T19:33:50.623Z] [2026-09-04T19:33:50.623Z] PLAY [test anvil package self update] ****************************************** [2026-09-04T19:33:50.623Z] [2026-09-04T19:33:50.623Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:33:53.706Z] ok: [an-a01n02] [2026-09-04T19:33:53.706Z] ok: [an-a01dr01] [2026-09-04T19:33:53.706Z] ok: [an-striker01] [2026-09-04T19:33:53.706Z] ok: [an-a01n01] [2026-09-04T19:33:53.706Z] ok: [an-striker02] [2026-09-04T19:33:53.706Z] [2026-09-04T19:33:53.706Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-09-04T19:33:57.579Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-09-04T19:33:57.579Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-09-04T19:33:57.579Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-09-04T19:33:57.579Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-09-04T19:33:57.579Z] changed: [an-a01n02] => (item=scancore.service) [2026-09-04T19:33:57.579Z] changed: [an-a01dr01] => (item=scancore.service) [2026-09-04T19:33:57.579Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-09-04T19:33:57.579Z] changed: [an-a01n01] => (item=scancore.service) [2026-09-04T19:33:57.579Z] changed: [an-striker01] => (item=scancore.service) [2026-09-04T19:33:57.579Z] changed: [an-striker02] => (item=scancore.service) [2026-09-04T19:33:57.579Z] [2026-09-04T19:33:57.579Z] PLAY [test striker package self update] **************************************** [2026-09-04T19:33:57.579Z] [2026-09-04T19:33:57.579Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:33:58.764Z] ok: [an-striker01] [2026-09-04T19:33:58.764Z] ok: [an-striker02] [2026-09-04T19:33:58.764Z] [2026-09-04T19:33:58.764Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-09-04T19:33:58.764Z] ] *** [2026-09-04T19:36:53.019Z] changed: [an-striker02] [2026-09-04T19:36:53.019Z] changed: [an-striker01] [2026-09-04T19:36:53.019Z] [2026-09-04T19:36:53.019Z] PLAY [test node package self update] ******************************************* [2026-09-04T19:36:53.019Z] [2026-09-04T19:36:53.019Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:36:53.019Z] ok: [an-a01n02] [2026-09-04T19:36:53.019Z] ok: [an-a01n01] [2026-09-04T19:36:53.019Z] [2026-09-04T19:36:53.019Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-09-04T19:36:53.019Z] ] *** [2026-09-04T19:37:41.362Z] changed: [an-a01n01] [2026-09-04T19:37:41.362Z] changed: [an-a01n02] [2026-09-04T19:37:41.362Z] [2026-09-04T19:37:41.362Z] PLAY [test dr package self update] ********************************************* [2026-09-04T19:37:41.362Z] [2026-09-04T19:37:41.362Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:37:41.362Z] ok: [an-a01dr01] [2026-09-04T19:37:41.362Z] [2026-09-04T19:37:41.362Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr [2026-09-04T19:37:41.362Z] ] *** [2026-09-04T19:38:29.909Z] changed: [an-a01dr01] [2026-09-04T19:38:29.909Z] [2026-09-04T19:38:29.909Z] PLAY [test anvil package self update] ****************************************** [2026-09-04T19:38:29.909Z] [2026-09-04T19:38:29.909Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:38:29.909Z] ok: [an-a01n02] [2026-09-04T19:38:29.909Z] ok: [an-a01dr01] [2026-09-04T19:38:29.909Z] ok: [an-a01n01] [2026-09-04T19:38:29.909Z] ok: [an-striker01] [2026-09-04T19:38:29.909Z] ok: [an-striker02] [2026-09-04T19:38:29.909Z] [2026-09-04T19:38:29.909Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-09-04T19:38:31.096Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-09-04T19:38:31.096Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-09-04T19:38:31.096Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-09-04T19:38:31.096Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-09-04T19:38:31.096Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-09-04T19:38:31.096Z] changed: [an-a01n02] => (item=scancore.service) [2026-09-04T19:38:31.096Z] changed: [an-a01dr01] => (item=scancore.service) [2026-09-04T19:38:31.096Z] changed: [an-a01n01] => (item=scancore.service) [2026-09-04T19:38:31.096Z] changed: [an-striker01] => (item=scancore.service) [2026-09-04T19:38:31.096Z] changed: [an-striker02] => (item=scancore.service) [2026-09-04T19:38:31.096Z] [2026-09-04T19:38:31.096Z] PLAY [delete server tests] ***************************************************** [2026-09-04T19:38:31.096Z] [2026-09-04T19:38:31.096Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:38:32.876Z] ok: [an-a01n02] [2026-09-04T19:38:32.876Z] ok: [an-a01n01] [2026-09-04T19:38:32.876Z] [2026-09-04T19:38:32.876Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-09-04T19:38:32.876Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-09-04T19:38:32.876Z] [2026-09-04T19:38:32.876Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-09-04T19:38:32.876Z] ] *** [2026-09-04T19:38:37.734Z] ok: [an-a01n02] => (item=an-a01n01) [2026-09-04T19:38:37.734Z] ok: [an-a01n01] => (item=an-a01n01) [2026-09-04T19:38:37.734Z] ok: [an-a01n02] => (item=an-a01n02) [2026-09-04T19:38:37.734Z] ok: [an-a01n01] => (item=an-a01n02) [2026-09-04T19:38:37.734Z] [2026-09-04T19:38:37.734Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-09-04T19:38:37.734Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-09-04T19:38:37.734Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-09-04T19:38:37.734Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-09-04T19:38:37.734Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-09-04T19:38:37.734Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-09-04T19:38:37.734Z] [2026-09-04T19:38:37.734Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:38:37.734Z] ] *** [2026-09-04T19:38:38.923Z] skipping: [an-a01n01] [2026-09-04T19:38:38.923Z] ok: [an-a01n02] [2026-09-04T19:38:38.923Z] [2026-09-04T19:38:38.923Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T19:38:38.923Z] --force \ [2026-09-04T19:38:38.923Z] --server an-test-deploy{{ servernum }} [2026-09-04T19:38:38.923Z] ] *** [2026-09-04T19:38:47.121Z] skipping: [an-a01n01] [2026-09-04T19:38:47.121Z] changed: [an-a01n02] [2026-09-04T19:38:47.121Z] [2026-09-04T19:38:47.121Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:38:47.121Z] ] *** [2026-09-04T19:39:35.476Z] skipping: [an-a01n01] [2026-09-04T19:39:35.476Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-09-04T19:39:35.476Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-09-04T19:39:35.476Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-09-04T19:39:35.476Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-09-04T19:39:35.476Z] ok: [an-a01n02] [2026-09-04T19:39:35.476Z] [2026-09-04T19:39:35.476Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T19:39:35.476Z] ] *** [2026-09-04T19:39:46.851Z] skipping: [an-a01n01] [2026-09-04T19:39:46.851Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-09-04T19:39:46.851Z] ok: [an-a01n02] [2026-09-04T19:39:46.851Z] [2026-09-04T19:39:46.851Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T19:39:46.851Z] ] *** [2026-09-04T19:39:46.851Z] skipping: [an-a01n01] [2026-09-04T19:39:46.851Z] ok: [an-a01n02] [2026-09-04T19:39:46.851Z] [2026-09-04T19:39:46.851Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:46.851Z] ] *** [2026-09-04T19:39:48.038Z] skipping: [an-a01n01] [2026-09-04T19:39:48.038Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.990372", "end": "2026-09-04 15:39:47.855512", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 15:39:46.865140", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T19:39:48.038Z] ...ignoring [2026-09-04T19:39:48.038Z] [2026-09-04T19:39:48.038Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T19:39:48.038Z] --force \ [2026-09-04T19:39:48.038Z] --server an-test-deploy{{ servernum }} [2026-09-04T19:39:48.038Z] ] *** [2026-09-04T19:39:48.038Z] skipping: [an-a01n01] [2026-09-04T19:39:48.038Z] skipping: [an-a01n02] [2026-09-04T19:39:48.038Z] [2026-09-04T19:39:48.038Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:48.038Z] ] *** [2026-09-04T19:39:48.380Z] skipping: [an-a01n01] [2026-09-04T19:39:48.380Z] skipping: [an-a01n02] [2026-09-04T19:39:48.380Z] [2026-09-04T19:39:48.380Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T19:39:48.380Z] ] *** [2026-09-04T19:39:48.380Z] skipping: [an-a01n01] [2026-09-04T19:39:48.380Z] skipping: [an-a01n02] [2026-09-04T19:39:48.380Z] [2026-09-04T19:39:48.380Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T19:39:48.380Z] ] *** [2026-09-04T19:39:48.380Z] skipping: [an-a01n01] [2026-09-04T19:39:48.380Z] skipping: [an-a01n02] [2026-09-04T19:39:48.380Z] [2026-09-04T19:39:48.380Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:48.380Z] ] *** [2026-09-04T19:39:50.167Z] skipping: [an-a01n01] [2026-09-04T19:39:50.167Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.987636", "end": "2026-09-04 15:39:49.761207", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 15:39:48.773571", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T19:39:50.167Z] ...ignoring [2026-09-04T19:39:50.167Z] [2026-09-04T19:39:50.167Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T19:39:50.167Z] --force \ [2026-09-04T19:39:50.167Z] --server an-test-deploy{{ servernum }} [2026-09-04T19:39:50.167Z] ] *** [2026-09-04T19:39:50.167Z] skipping: [an-a01n01] [2026-09-04T19:39:50.167Z] skipping: [an-a01n02] [2026-09-04T19:39:50.167Z] [2026-09-04T19:39:50.167Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:50.167Z] ] *** [2026-09-04T19:39:50.167Z] skipping: [an-a01n01] [2026-09-04T19:39:50.167Z] skipping: [an-a01n02] [2026-09-04T19:39:50.167Z] [2026-09-04T19:39:50.167Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T19:39:50.167Z] ] *** [2026-09-04T19:39:50.167Z] skipping: [an-a01n01] [2026-09-04T19:39:50.167Z] skipping: [an-a01n02] [2026-09-04T19:39:50.167Z] [2026-09-04T19:39:50.167Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T19:39:50.167Z] ] *** [2026-09-04T19:39:50.167Z] skipping: [an-a01n01] [2026-09-04T19:39:50.167Z] skipping: [an-a01n02] [2026-09-04T19:39:50.167Z] [2026-09-04T19:39:50.167Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:50.167Z] ] *** [2026-09-04T19:39:51.879Z] skipping: [an-a01n01] [2026-09-04T19:39:51.879Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.948096", "end": "2026-09-04 15:39:51.626463", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 15:39:50.678367", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T19:39:51.879Z] ...ignoring [2026-09-04T19:39:51.879Z] [2026-09-04T19:39:51.879Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T19:39:51.879Z] --force \ [2026-09-04T19:39:51.879Z] --server an-test-deploy{{ servernum }} [2026-09-04T19:39:51.879Z] ] *** [2026-09-04T19:39:51.879Z] skipping: [an-a01n01] [2026-09-04T19:39:51.879Z] skipping: [an-a01n02] [2026-09-04T19:39:51.879Z] [2026-09-04T19:39:51.879Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:51.879Z] ] *** [2026-09-04T19:39:52.226Z] skipping: [an-a01n01] [2026-09-04T19:39:52.226Z] skipping: [an-a01n02] [2026-09-04T19:39:52.226Z] [2026-09-04T19:39:52.226Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T19:39:52.226Z] ] *** [2026-09-04T19:39:52.226Z] skipping: [an-a01n01] [2026-09-04T19:39:52.226Z] skipping: [an-a01n02] [2026-09-04T19:39:52.226Z] [2026-09-04T19:39:52.226Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T19:39:52.226Z] ] *** [2026-09-04T19:39:52.226Z] skipping: [an-a01n01] [2026-09-04T19:39:52.226Z] skipping: [an-a01n02] [2026-09-04T19:39:52.226Z] [2026-09-04T19:39:52.226Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:52.226Z] ] *** [2026-09-04T19:39:53.938Z] skipping: [an-a01n01] [2026-09-04T19:39:53.938Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.032746", "end": "2026-09-04 15:39:53.568976", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 15:39:52.536230", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-09-04T19:39:53.938Z] ...ignoring [2026-09-04T19:39:53.938Z] [2026-09-04T19:39:53.938Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-09-04T19:39:53.938Z] --force \ [2026-09-04T19:39:53.938Z] --server an-test-deploy{{ servernum }} [2026-09-04T19:39:53.938Z] ] *** [2026-09-04T19:39:53.938Z] skipping: [an-a01n01] [2026-09-04T19:39:53.938Z] skipping: [an-a01n02] [2026-09-04T19:39:53.938Z] [2026-09-04T19:39:53.938Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-09-04T19:39:53.938Z] ] *** [2026-09-04T19:39:53.938Z] skipping: [an-a01n01] [2026-09-04T19:39:53.938Z] skipping: [an-a01n02] [2026-09-04T19:39:53.938Z] [2026-09-04T19:39:53.938Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-09-04T19:39:53.938Z] ] *** [2026-09-04T19:39:53.938Z] skipping: [an-a01n01] [2026-09-04T19:39:53.938Z] skipping: [an-a01n02] [2026-09-04T19:39:53.938Z] [2026-09-04T19:39:53.938Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-09-04T19:39:53.938Z] ] *** [2026-09-04T19:39:54.281Z] skipping: [an-a01n01] [2026-09-04T19:39:54.281Z] skipping: [an-a01n02] [2026-09-04T19:39:54.281Z] [2026-09-04T19:39:54.281Z] PLAY [Check anvil available resources from striker01] ************************** [2026-09-04T19:39:54.281Z] [2026-09-04T19:39:54.281Z] TASK [Gathering Facts ] ******************************************************** [2026-09-04T19:39:55.993Z] ok: [an-striker01] [2026-09-04T19:39:55.993Z] [2026-09-04T19:39:55.993Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-09-04T19:39:55.993Z] sleep 300 [2026-09-04T19:39:55.993Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-09-04T19:39:55.993Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-09-04T19:39:55.993Z] ] *** [2026-09-04T19:45:06.608Z] fatal: [an-striker01]: FAILED! => {"changed": false, "cmd": "echo \"Waiting 5 minutes for scancore to pick up latest status\"\nsleep 300\nanvil-report-usage > /root/anvil_available_resources.stop\ndiff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop\n", "delta": "0:05:02.766368", "end": "2026-09-04 15:44:59.407025", "msg": "non-zero return code", "rc": 1, "start": "2026-09-04 15:39:56.640657", "stderr": "", "stderr_lines": [], "stdout": "Waiting 5 minutes for scancore to pick up latest status\n--- /root/anvil_available_resources.start\t2026-09-04 13:18:38.989411144 -0400\n+++ /root/anvil_available_resources.stop\t2026-09-04 15:44:59.322129853 -0400\n@@ -7,5 +7,5 @@\n +--------------+---------+-----------+----------------------------+------------------------+------------+\n | Anvil! Nodes | CPU | RAM Free | Bridges | Storage Group | Free |\n +--------------+---------+-----------+----------------------------+------------------------+------------+\n-| an-anvil-01 | 8c (8t) | 24.00 GiB | bcn1_bridge1, ifn1_bridge1 | Node an-anvil-01, SG 1 | 200.00 GiB |\n+| an-anvil-01 | 8c (8t) | 24.00 GiB | bcn1_bridge1, ifn1_bridge1 | Node an-anvil-01, SG 1 | 190.00 GiB |\n +--------------+---------+-----------+----------------------------+------------------------+------------+", "stdout_lines": ["Waiting 5 minutes for scancore to pick up latest status", "--- /root/anvil_available_resources.start\t2026-09-04 13:18:38.989411144 -0400", "+++ /root/anvil_available_resources.stop\t2026-09-04 15:44:59.322129853 -0400", "@@ -7,5 +7,5 @@", " +--------------+---------+-----------+----------------------------+------------------------+------------+", " | Anvil! Nodes | CPU | RAM Free | Bridges | Storage Group | Free |", " +--------------+---------+-----------+----------------------------+------------------------+------------+", "-| an-anvil-01 | 8c (8t) | 24.00 GiB | bcn1_bridge1, ifn1_bridge1 | Node an-anvil-01, SG 1 | 200.00 GiB |", "+| an-anvil-01 | 8c (8t) | 24.00 GiB | bcn1_bridge1, ifn1_bridge1 | Node an-anvil-01, SG 1 | 190.00 GiB |", " +--------------+---------+-----------+----------------------------+------------------------+------------+"]} [2026-09-04T19:45:06.608Z] [2026-09-04T19:45:06.608Z] NO MORE HOSTS LEFT ************************************************************* [2026-09-04T19:45:06.608Z] [2026-09-04T19:45:06.608Z] PLAY RECAP ********************************************************************* [2026-09-04T19:45:06.608Z] an-a01dr01 : ok=90 changed=38 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [2026-09-04T19:45:06.608Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-09-04T19:45:06.608Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-09-04T19:45:06.608Z] an-striker01 : ok=56 changed=19 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 [2026-09-04T19:45:06.608Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-09-04T19:45:06.608Z] [2026-09-04T19:45:06.608Z] tar: Removing leading `/' from member names [2026-09-04T19:45:06.608Z] Data collection has begun. This will take a while! [2026-09-04T19:45:06.608Z] [2026-09-04T19:45:06.608Z] Preparing local machine [2026-09-04T19:45:06.608Z] - Output directory [/root] already exists. [2026-09-04T19:45:06.608Z] - Creating temporary data dir [/tmp/anvil-debug_2026-09-04_15-45-04]... Done! [2026-09-04T19:45:06.608Z] [2026-09-04T19:45:06.608Z] Grabbing logs and data from the local system. [2026-09-04T19:45:06.608Z] - Grabbing system logs from the previous boot... Done! [2026-09-04T19:45:06.608Z] - Grabbing system logs from this boot... Done! [2026-09-04T19:45:08.927Z] - Dumping and compressing database data, PLEASE BE PATIENT!... Done! [2026-09-04T19:45:08.927Z] - Screehshots collection disabled. [2026-09-04T19:45:08.927Z] - Grabbing hosts file... Done! [2026-09-04T19:45:08.927Z] - Grabbing Anvil! config... Done! [2026-09-04T19:45:08.927Z] - Grabbing Anvil! log... Done! [2026-09-04T19:45:08.927Z] - Grabbing cloud-init logs... Done! [2026-09-04T19:45:08.927Z] [2026-09-04T19:45:08.927Z] Grabbing logs and data from the remote system: [an-striker02]. [2026-09-04T19:45:08.927Z] - Testing access... [2026-09-04T19:45:10.103Z] - Found access over the network: [bcn1] using the target IP: [10.201.4.2] [2026-09-04T19:45:10.573Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.3] [2026-09-04T19:45:12.494Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T19:45:13.570Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T19:45:14.305Z] - Grabbing cloud-init logs... Done. [2026-09-04T19:45:16.017Z] - Dumping and compressing remote database data, PLEASE BE PATIENT!... Done! [2026-09-04T19:45:16.358Z] - Grabbing hosts file... Done. [2026-09-04T19:45:49.939Z] - Grabbing Anvil! log... Done. [2026-09-04T19:45:49.939Z] - Grabbing audit logs... Done. [2026-09-04T19:45:49.939Z] - Grabbing Anvil! config... Done. [2026-09-04T19:45:49.939Z] [2026-09-04T19:45:49.939Z] Grabbing logs and data from the remote system: [an-a01n01]. [2026-09-04T19:45:49.939Z] - Testing access... [2026-09-04T19:45:50.282Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.1] [2026-09-04T19:45:50.888Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.4] [2026-09-04T19:45:51.574Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T19:45:52.259Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T19:45:52.602Z] - Grabbing cloud-init logs... Done. [2026-09-04T19:45:52.944Z] - Grabbing hosts file... Done. [2026-09-04T19:46:50.934Z] - Grabbing Anvil! log... Done. [2026-09-04T19:46:50.934Z] - Grabbing audit logs... Done. [2026-09-04T19:46:50.934Z] - Grabbing Anvil! config... Done. [2026-09-04T19:46:51.276Z] - Collecting the cluster information base (CIB)... Done! Copying to here... Done. [2026-09-04T19:46:51.617Z] - Collecting the corosync config... Done. [2026-09-04T19:46:51.959Z] - Collecting server definitions... Done! [2026-09-04T19:46:52.301Z] - Collecting replicated storage config... Done! [2026-09-04T19:46:52.301Z] [2026-09-04T19:46:52.301Z] Grabbing logs and data from the remote system: [an-a01n02]. [2026-09-04T19:46:52.301Z] - Testing access... [2026-09-04T19:46:53.035Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.2] [2026-09-04T19:46:53.576Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.5] [2026-09-04T19:46:54.324Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T19:46:55.008Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T19:46:55.350Z] - Grabbing cloud-init logs... Done. [2026-09-04T19:46:55.694Z] - Grabbing hosts file... Done. [2026-09-04T19:47:53.715Z] - Grabbing Anvil! log... Done. [2026-09-04T19:47:53.715Z] - Grabbing audit logs... Done. [2026-09-04T19:47:53.715Z] - Grabbing Anvil! config... Done. [2026-09-04T19:47:53.715Z] - Collecting the cluster information base (CIB)... Done! Copying to here... Done. [2026-09-04T19:47:53.715Z] - Collecting the corosync config... Done. [2026-09-04T19:47:53.715Z] - Collecting server definitions... Done! [2026-09-04T19:47:53.715Z] - Collecting replicated storage config... Done! [2026-09-04T19:47:53.715Z] [2026-09-04T19:47:53.715Z] Grabbing logs and data from the remote system: [an-a01dr01]. [2026-09-04T19:47:53.715Z] - Testing access... [2026-09-04T19:47:53.715Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.3] [2026-09-04T19:47:53.715Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.6] [2026-09-04T19:47:53.715Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T19:47:53.715Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T19:47:53.715Z] - Grabbing cloud-init logs... Done. [2026-09-04T19:47:53.715Z] - Grabbing hosts file... Done. [2026-09-04T19:48:27.293Z] - Grabbing Anvil! log... Done. [2026-09-04T19:48:27.293Z] - Grabbing audit logs... Done. [2026-09-04T19:48:27.293Z] - Grabbing Anvil! config... Done. [2026-09-04T19:48:27.293Z] - Collecting server definitions... Done! [2026-09-04T19:48:27.293Z] - Collecting replicated storage config... Done! [2026-09-04T19:48:27.293Z] [2026-09-04T19:48:27.293Z] Data collection complete [2026-09-04T20:26:20.540Z] - Creating the tarball now. PLEASE BE PATIENT!... Done! [2026-09-04T20:26:20.540Z] - Removing temporary data dir [/tmp/anvil-debug_2026-09-04_15-45-04]... Done! [2026-09-04T20:26:20.540Z] [2026-09-04T20:26:20.540Z] [ Complete ] - The debug data collected here: [/root/debug.tar.bz2] [2026-09-04T20:26:20.540Z] [ Warning ] - The collected data and logs likely include sensitive information! Share it carefully! [2026-09-04T20:26:24.409Z] Data collection has begun. This will take a while! [2026-09-04T20:26:24.409Z] [2026-09-04T20:26:24.409Z] Preparing local machine [2026-09-04T20:26:24.409Z] - Output directory [/root] already exists. [2026-09-04T20:26:24.409Z] - Creating temporary data dir [/tmp/anvil-debug_2026-09-04_16-26-24]... Done! [2026-09-04T20:26:24.409Z] [2026-09-04T20:26:24.409Z] Grabbing logs and data from the remote system: [an-striker01]. [2026-09-04T20:26:24.409Z] - Testing access... [2026-09-04T20:26:25.144Z] - Found access over the network: [bcn1] using the target IP: [10.201.4.1] [2026-09-04T20:26:25.487Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.2] [2026-09-04T20:26:28.148Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T20:26:29.223Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T20:26:29.957Z] - Grabbing cloud-init logs... Done. [2026-09-04T20:26:31.668Z] - Dumping and compressing remote database data, PLEASE BE PATIENT!... Done! [2026-09-04T20:26:32.011Z] - Grabbing hosts file... Done. [2026-09-04T20:27:12.665Z] - Grabbing Anvil! log... Done. [2026-09-04T20:27:13.008Z] - Grabbing audit logs... Done. [2026-09-04T20:27:13.743Z] - Grabbing Anvil! config... Done. [2026-09-04T20:27:13.743Z] [2026-09-04T20:27:13.743Z] Grabbing logs and data from the local system. [2026-09-04T20:27:15.453Z] - Grabbing system logs from the previous boot... Done! [2026-09-04T20:27:16.640Z] - Grabbing system logs from this boot... Done! [2026-09-04T20:27:19.737Z] - Dumping and compressing database data, PLEASE BE PATIENT!... Done! [2026-09-04T20:27:19.737Z] - Screehshots collection disabled. [2026-09-04T20:27:19.737Z] - Grabbing hosts file... Done! [2026-09-04T20:27:19.737Z] - Grabbing Anvil! config... Done! [2026-09-04T20:27:19.737Z] - Grabbing Anvil! log... Done! [2026-09-04T20:27:19.737Z] - Grabbing cloud-init logs... Done! [2026-09-04T20:27:19.737Z] [2026-09-04T20:27:19.737Z] Grabbing logs and data from the remote system: [an-a01n01]. [2026-09-04T20:27:19.737Z] - Testing access... [2026-09-04T20:27:21.022Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.1] [2026-09-04T20:27:21.022Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.4] [2026-09-04T20:27:21.708Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T20:27:22.392Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T20:27:23.076Z] - Grabbing cloud-init logs... Done. [2026-09-04T20:27:23.419Z] - Grabbing hosts file... Done. [2026-09-04T20:28:33.070Z] - Grabbing Anvil! log... Done. [2026-09-04T20:28:33.070Z] - Grabbing audit logs... Done. [2026-09-04T20:28:33.070Z] - Grabbing Anvil! config... Done. [2026-09-04T20:28:33.070Z] - Collecting the cluster information base (CIB)... Done! Copying to here... Done. [2026-09-04T20:28:33.070Z] - Collecting the corosync config... Done. [2026-09-04T20:28:33.070Z] - Collecting server definitions... Done! [2026-09-04T20:28:33.070Z] - Collecting replicated storage config... Done! [2026-09-04T20:28:33.070Z] [2026-09-04T20:28:33.070Z] Grabbing logs and data from the remote system: [an-a01n02]. [2026-09-04T20:28:33.070Z] - Testing access... [2026-09-04T20:28:33.070Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.2] [2026-09-04T20:28:33.070Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.5] [2026-09-04T20:28:34.146Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T20:28:35.472Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T20:28:36.137Z] - Grabbing cloud-init logs... Done. [2026-09-04T20:28:36.137Z] - Grabbing hosts file... Done. [2026-09-04T20:29:45.760Z] - Grabbing Anvil! log... Done. [2026-09-04T20:29:45.760Z] - Grabbing audit logs... Done. [2026-09-04T20:29:45.760Z] - Grabbing Anvil! config... Done. [2026-09-04T20:29:46.918Z] - Collecting the cluster information base (CIB)... Done! Copying to here... Done. [2026-09-04T20:29:47.264Z] - Collecting the corosync config... Done. [2026-09-04T20:29:47.648Z] - Collecting server definitions... Done! [2026-09-04T20:29:47.993Z] - Collecting replicated storage config... Done! [2026-09-04T20:29:47.993Z] [2026-09-04T20:29:47.993Z] Grabbing logs and data from the remote system: [an-a01dr01]. [2026-09-04T20:29:47.993Z] - Testing access... [2026-09-04T20:29:48.602Z] - Found access over the network: [bcn1] using the target IP: [10.201.10.3] [2026-09-04T20:29:49.347Z] - Found access over the network: [ifn1] using the target IP: [192.168.122.6] [2026-09-04T20:29:53.471Z] - Writing out system logs from the previous boot... Done! Copying to here... Done. [2026-09-04T20:29:54.435Z] - Grabbing system logs from this boot... Done! Copying to here... Done. [2026-09-04T20:29:54.435Z] - Grabbing cloud-init logs... Done. [2026-09-04T20:29:54.435Z] - Grabbing hosts file... Done. [2026-09-04T20:30:42.854Z] - Grabbing Anvil! log... Done. [2026-09-04T20:30:42.854Z] - Grabbing audit logs... Done. [2026-09-04T20:30:42.854Z] - Grabbing Anvil! config... Done. [2026-09-04T20:30:42.854Z] - Collecting server definitions... Done! [2026-09-04T20:30:42.854Z] - Collecting replicated storage config... Done! [2026-09-04T20:30:42.854Z] [2026-09-04T20:30:42.854Z] Data collection complete [2026-09-04T21:21:31.838Z] - Creating the tarball now. PLEASE BE PATIENT!... Done! [2026-09-04T21:21:31.838Z] - Removing temporary data dir [/tmp/anvil-debug_2026-09-04_16-26-24]... Done! [2026-09-04T21:21:31.838Z] [2026-09-04T21:21:31.838Z] [ Complete ] - The debug data collected here: [/root/debug.tar.bz2] [2026-09-04T21:21:31.838Z] [ Warning ] - The collected data and logs likely include sensitive information! Share it carefully! [2026-09-04T21:21:51.174Z] Will stop any running daemons ('--disable --now' used) after disabling daemons. [2026-09-04T21:21:51.174Z] Disabling: [anvil-daemon] now... [2026-09-04T21:21:51.174Z] Stopping: [anvil-daemon] now... [2026-09-04T21:21:51.174Z] Disabling: [anvil-monitor-network] now... [2026-09-04T21:21:51.174Z] Stopping: [anvil-monitor-network] now... [2026-09-04T21:21:51.174Z] Disabling: [scancore] now... [2026-09-04T21:21:51.174Z] Stopping: [scancore] now... [2026-09-04T21:21:51.174Z] Disabling: [striker-ui-api] now... [2026-09-04T21:21:51.174Z] Stopping: [striker-ui-api] now... [2026-09-04T21:21:51.174Z] Done. [2026-09-04T21:21:51.174Z] [2026-09-04T21:21:52.359Z] Anvil! Intelligent Availability Daemon Status (Enabled, Started, Failed); [2026-09-04T21:21:52.359Z] Daemons [enabled/started/failed]; anvil-daemon [Disabled/Stopped/OK], anvil-monitor-daemons [Disabled/Stopped/OK], anvil-monitor-lvm [Disabled/Stopped/OK], anvil-monitor-network [Disabled/Stopped/OK], anvil-monitor-performance [Disabled/Stopped/OK], scancore [Disabled/Stopped/OK], striker-ui-api [Disabled/Stopped/OK]. [2026-09-04T21:21:55.393Z] Will stop any running daemons ('--disable --now' used) after disabling daemons. [2026-09-04T21:21:55.393Z] Disabling: [anvil-daemon] now... [2026-09-04T21:21:55.737Z] Stopping: [anvil-daemon] now... [2026-09-04T21:21:56.085Z] Disabling: [anvil-monitor-network] now... [2026-09-04T21:21:56.428Z] Stopping: [anvil-monitor-network] now... [2026-09-04T21:21:56.772Z] Disabling: [scancore] now... [2026-09-04T21:21:57.116Z] Stopping: [scancore] now... [2026-09-04T21:21:57.464Z] Disabling: [striker-ui-api] now... [2026-09-04T21:21:57.810Z] Stopping: [striker-ui-api] now... [2026-09-04T21:21:57.810Z] Done. [2026-09-04T21:21:57.810Z] [2026-09-04T21:21:59.304Z] Anvil! Intelligent Availability Daemon Status (Enabled, Started, Failed); [2026-09-04T21:21:59.304Z] Daemons [enabled/started/failed]; anvil-daemon [Disabled/Stopped/OK], anvil-monitor-daemons [Disabled/Stopped/OK], anvil-monitor-lvm [Disabled/Stopped/OK], anvil-monitor-network [Disabled/Stopped/OK], anvil-monitor-performance [Disabled/Stopped/OK], scancore [Disabled/Stopped/OK], striker-ui-api [Disabled/Stopped/OK]. [2026-09-04T21:22:01.740Z] Will stop any running daemons ('--disable --now' used) after disabling daemons. [2026-09-04T21:22:01.740Z] Disabling: [anvil-daemon] now... [2026-09-04T21:22:02.168Z] Stopping: [anvil-daemon] now... [2026-09-04T21:22:02.168Z] Disabling: [anvil-monitor-network] now... [2026-09-04T21:22:02.516Z] Stopping: [anvil-monitor-network] now... [2026-09-04T21:22:02.864Z] Disabling: [anvil-safe-start] now... [2026-09-04T21:22:03.293Z] Stopping: [anvil-safe-start] now... [2026-09-04T21:22:03.293Z] Disabling: [scancore] now... [2026-09-04T21:22:03.642Z] Stopping: [scancore] now... [2026-09-04T21:22:03.642Z] Done. [2026-09-04T21:22:03.642Z] [2026-09-04T21:22:05.365Z] Anvil! Intelligent Availability Daemon Status (Enabled, Started, Failed); [2026-09-04T21:22:05.365Z] Daemons [enabled/started/failed]; anvil-daemon [Disabled/Stopped/OK], anvil-monitor-daemons [Disabled/Stopped/OK], anvil-monitor-lvm [Disabled/Stopped/OK], anvil-monitor-network [Disabled/Stopped/OK], anvil-monitor-performance [Disabled/Stopped/OK], anvil-safe-start [Disabled/Stopped/OK], scancore [Disabled/Stopped/OK]. [2026-09-04T21:22:07.074Z] Will stop any running daemons ('--disable --now' used) after disabling daemons. [2026-09-04T21:22:07.074Z] Disabling: [anvil-daemon] now... [2026-09-04T21:22:07.417Z] Stopping: [anvil-daemon] now... [2026-09-04T21:22:07.759Z] Disabling: [anvil-monitor-network] now... [2026-09-04T21:22:08.102Z] Stopping: [anvil-monitor-network] now... [2026-09-04T21:22:08.102Z] Disabling: [anvil-safe-start] now... [2026-09-04T21:22:08.443Z] Stopping: [anvil-safe-start] now... [2026-09-04T21:22:08.443Z] Disabling: [scancore] now... [2026-09-04T21:22:08.785Z] Stopping: [scancore] now... [2026-09-04T21:22:08.785Z] Done. [2026-09-04T21:22:08.785Z] [2026-09-04T21:22:10.500Z] Anvil! Intelligent Availability Daemon Status (Enabled, Started, Failed); [2026-09-04T21:22:10.500Z] Daemons [enabled/started/failed]; anvil-daemon [Disabled/Stopped/OK], anvil-monitor-daemons [Disabled/Stopped/OK], anvil-monitor-lvm [Disabled/Stopped/OK], anvil-monitor-network [Disabled/Stopped/OK], anvil-monitor-performance [Disabled/Stopped/OK], anvil-safe-start [Disabled/Stopped/OK], scancore [Disabled/Stopped/OK]. [2026-09-04T21:22:12.213Z] Will stop any running daemons ('--disable --now' used) after disabling daemons. [2026-09-04T21:22:12.213Z] Disabling: [anvil-daemon] now... [2026-09-04T21:22:13.235Z] Stopping: [anvil-daemon] now... [2026-09-04T21:22:13.235Z] Disabling: [anvil-monitor-network] now... [2026-09-04T21:22:13.708Z] Stopping: [anvil-monitor-network] now... [2026-09-04T21:22:13.708Z] Disabling: [scancore] now... [2026-09-04T21:22:13.708Z] Stopping: [scancore] now... [2026-09-04T21:22:13.708Z] Done. [2026-09-04T21:22:13.708Z] [2026-09-04T21:22:14.443Z] Anvil! Intelligent Availability Daemon Status (Enabled, Started, Failed); [2026-09-04T21:22:14.443Z] Daemons [enabled/started/failed]; anvil-daemon [Disabled/Stopped/OK], anvil-monitor-daemons [Disabled/Stopped/OK], anvil-monitor-lvm [Disabled/Stopped/OK], anvil-monitor-network [Disabled/Stopped/OK], anvil-monitor-performance [Disabled/Stopped/OK], anvil-safe-start [Disabled/Stopped/OK], scancore [Disabled/Stopped/OK]. [2026-09-04T21:23:02.788Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-09-04T21:23:06.683Z] /root/ci-tools [2026-09-04T21:23:06.683Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/1130.tar.gz [2026-09-04T21:23:07.025Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-09-04T21:23:07.025Z] This might take a few seconds. [2026-09-04T21:23:07.025Z] [2026-09-04T21:23:07.025Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T21:23:08.734Z] 𐄂 Disconnected from Red Hat Insights [2026-09-04T21:23:09.590Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-09-04T21:23:09.590Z] [2026-09-04T21:23:09.590Z] Manage your connected systems: https://red.ht/connector [2026-09-04T21:23:10.776Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T21:23:10.776Z] [2026-09-04T21:23:10.776Z] 1. Register with subscription-manager [2026-09-04T21:23:10.776Z] # subscription-manager register [2026-09-04T21:23:10.776Z] [2026-09-04T21:23:10.776Z] 2. Register with insights-client [2026-09-04T21:23:10.776Z] # insights-client --register [2026-09-04T21:23:11.964Z] This system is currently not registered. [2026-09-04T21:23:12.317Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-09-04T21:23:12.317Z] This might take a few seconds. [2026-09-04T21:23:12.317Z] [2026-09-04T21:23:12.317Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T21:23:14.028Z] 𐄂 Disconnected from Red Hat Insights [2026-09-04T21:23:14.763Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-09-04T21:23:14.763Z] [2026-09-04T21:23:14.763Z] Manage your connected systems: https://red.ht/connector [2026-09-04T21:23:15.947Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T21:23:15.947Z] [2026-09-04T21:23:15.947Z] 1. Register with subscription-manager [2026-09-04T21:23:15.947Z] # subscription-manager register [2026-09-04T21:23:15.947Z] [2026-09-04T21:23:15.947Z] 2. Register with insights-client [2026-09-04T21:23:15.947Z] # insights-client --register [2026-09-04T21:23:17.134Z] This system is currently not registered. [2026-09-04T21:23:17.476Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-09-04T21:23:17.476Z] This might take a few seconds. [2026-09-04T21:23:17.476Z] [2026-09-04T21:23:17.476Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T21:23:19.188Z] 𐄂 Disconnected from Red Hat Insights [2026-09-04T21:23:19.922Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-09-04T21:23:19.922Z] [2026-09-04T21:23:19.922Z] Manage your connected systems: https://red.ht/connector [2026-09-04T21:23:21.631Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T21:23:21.631Z] [2026-09-04T21:23:21.631Z] 1. Register with subscription-manager [2026-09-04T21:23:21.631Z] # subscription-manager register [2026-09-04T21:23:21.631Z] [2026-09-04T21:23:21.631Z] 2. Register with insights-client [2026-09-04T21:23:21.631Z] # insights-client --register [2026-09-04T21:23:22.367Z] This system is currently not registered. [2026-09-04T21:23:22.711Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-09-04T21:23:22.711Z] This might take a few seconds. [2026-09-04T21:23:22.711Z] [2026-09-04T21:23:22.711Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T21:23:24.421Z] 𐄂 Disconnected from Red Hat Insights [2026-09-04T21:23:25.159Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-09-04T21:23:25.159Z] [2026-09-04T21:23:25.159Z] Manage your connected systems: https://red.ht/connector [2026-09-04T21:23:26.346Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T21:23:26.346Z] [2026-09-04T21:23:26.346Z] 1. Register with subscription-manager [2026-09-04T21:23:26.346Z] # subscription-manager register [2026-09-04T21:23:26.346Z] [2026-09-04T21:23:26.346Z] 2. Register with insights-client [2026-09-04T21:23:26.346Z] # insights-client --register [2026-09-04T21:23:27.531Z] This system is currently not registered. [2026-09-04T21:23:27.873Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-09-04T21:23:27.873Z] This might take a few seconds. [2026-09-04T21:23:27.873Z] [2026-09-04T21:23:27.873Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-09-04T21:23:29.582Z] 𐄂 Disconnected from Red Hat Insights [2026-09-04T21:23:30.316Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-09-04T21:23:30.316Z] [2026-09-04T21:23:30.316Z] Manage your connected systems: https://red.ht/connector [2026-09-04T21:23:31.501Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-09-04T21:23:31.501Z] [2026-09-04T21:23:31.501Z] 1. Register with subscription-manager [2026-09-04T21:23:31.501Z] # subscription-manager register [2026-09-04T21:23:31.501Z] [2026-09-04T21:23:31.501Z] 2. Register with insights-client [2026-09-04T21:23:31.501Z] # insights-client --register [2026-09-04T21:23:32.686Z] This system is currently not registered. [2026-09-04T21:23:32.761Z] [Pipeline] } [2026-09-04T21:23:32.764Z] [Pipeline] // timeout [2026-09-04T21:23:32.767Z] [Pipeline] echo [2026-09-04T21:23:32.767Z] runWithTimeout caught AbortException hudson.AbortException: script returned exit code 2 [2026-09-04T21:23:32.768Z] [Pipeline] echo [2026-09-04T21:23:32.768Z] runWithTimeout: script exit non-zero [2026-09-04T21:23:32.772Z] [Pipeline] echo [2026-09-04T21:23:32.772Z] RC runWithTimeout returned ERROR [2026-09-04T21:23:32.783Z] [Pipeline] sh [2026-09-04T21:23:33.391Z] [Pipeline] } [2026-09-04T21:23:33.394Z] [Pipeline] // script [2026-09-04T21:23:33.397Z] [Pipeline] } [2026-09-04T21:23:33.399Z] [Pipeline] // stage [2026-09-04T21:23:33.402Z] [Pipeline] stage [2026-09-04T21:23:33.402Z] [Pipeline] { (Declarative: Post Actions) [2026-09-04T21:23:33.454Z] [Pipeline] cleanWs [2026-09-04T21:23:33.504Z] [WS-CLEANUP] Deleting project workspace... [2026-09-04T21:23:33.504Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-09-04T21:23:33.528Z] [WS-CLEANUP] done [2026-09-04T21:23:33.532Z] [Pipeline] script [2026-09-04T21:23:33.534Z] [Pipeline] { [2026-09-04T21:23:33.548Z] [Pipeline] stage [2026-09-04T21:23:33.549Z] [Pipeline] { (Copying logs) [2026-09-04T21:23:33.628Z] [Pipeline] node [2026-09-04T21:23:33.741Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-09-04T21:23:33.741Z] [Pipeline] { [2026-09-04T21:23:33.779Z] [Pipeline] node [2026-09-04T21:23:33.882Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-09-04T21:23:33.882Z] [Pipeline] { [2026-09-04T21:23:33.886Z] [Pipeline] sh [2026-09-04T21:23:34.142Z] + mkdir -p /var/lib/jenkins/locks [2026-09-04T21:23:34.151Z] [Pipeline] echo [2026-09-04T21:23:34.151Z] RWLock: FD for lock log_archive in stage projectFinishUp is 389 [2026-09-04T21:23:34.154Z] [Pipeline] sleep [2026-09-04T21:23:34.154Z] No need to sleep any longer [2026-09-04T21:23:34.157Z] [Pipeline] echo [2026-09-04T21:23:34.157Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-09-04T21:23:34.160Z] [Pipeline] } [2026-09-04T21:23:34.162Z] [Pipeline] // node [2026-09-04T21:23:34.165Z] [Pipeline] echo [2026-09-04T21:23:34.165Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1130/ [2026-09-04T21:23:34.166Z] [Pipeline] sh [2026-09-04T21:23:34.422Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1130/ [2026-09-04T21:23:34.423Z] [Pipeline] sh