[2026-04-16T05:52:00.132Z] Started by upstream project "anvil/anvil-pipeline/PR-1000" build number 1 [2026-04-16T05:52:00.132Z] originally caused by: [2026-04-16T05:52:00.132Z] Pull request #1000 opened [2026-04-16T05:52:00.597Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-04-16T05:52:00.606Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-04-16T05:52:00.774Z] Examining kronosnet/ci-tools [2026-04-16T05:52:00.774Z] Attempting to resolve main as a branch [2026-04-16T05:52:00.929Z] Resolved main as branch main at revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 [2026-04-16T05:52:00.930Z] Selected Git installation does not exist. Using Default [2026-04-16T05:52:00.930Z] The recommended git tool is: NONE [2026-04-16T05:52:00.930Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-16T05:52:00.934Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-16T05:52:00.934Z] Wiping out workspace first. [2026-04-16T05:52:00.956Z] Cloning the remote Git repository [2026-04-16T05:52:00.956Z] Using shallow clone with depth 1 [2026-04-16T05:52:00.956Z] Avoid fetching tags [2026-04-16T05:52:00.956Z] Honoring refspec on initial clone [2026-04-16T05:52:00.956Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-16T05:52:00.956Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-04-16T05:52:00.966Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-16T05:52:00.966Z] > git --version # timeout=10 [2026-04-16T05:52:00.973Z] > git --version # 'git version 2.47.3' [2026-04-16T05:52:00.973Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-16T05:52:00.979Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T05:52:01.182Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-16T05:52:01.189Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T05:52:01.197Z] Avoid second fetch [2026-04-16T05:52:01.197Z] Checking out Revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 (main) [2026-04-16T05:52:01.197Z] > git config core.sparsecheckout # timeout=10 [2026-04-16T05:52:01.203Z] > git checkout -f 73ed6275b1ab90dc4db382514cbc5070a56e2e90 # timeout=10 [2026-04-16T05:52:01.234Z] Commit message: "Revert "New getUpNodesByLabel() call"" [2026-04-16T05:52:01.240Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-04-16T05:52:01.415Z] Examining kronosnet/ci-tools [2026-04-16T05:52:01.415Z] Attempting to resolve main as a branch [2026-04-16T05:52:01.566Z] Resolved main as branch main at revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 [2026-04-16T05:52:01.566Z] Selected Git installation does not exist. Using Default [2026-04-16T05:52:01.566Z] The recommended git tool is: NONE [2026-04-16T05:52:01.567Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-16T05:52:01.570Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-16T05:52:01.570Z] Wiping out workspace first. [2026-04-16T05:52:01.592Z] Cloning the remote Git repository [2026-04-16T05:52:01.592Z] Using shallow clone with depth 1 [2026-04-16T05:52:01.592Z] Avoid fetching tags [2026-04-16T05:52:01.592Z] Honoring refspec on initial clone [2026-04-16T05:52:01.592Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-16T05:52:01.592Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@5 # timeout=10 [2026-04-16T05:52:01.603Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-16T05:52:01.603Z] > git --version # timeout=10 [2026-04-16T05:52:01.610Z] > git --version # 'git version 2.47.3' [2026-04-16T05:52:01.610Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-16T05:52:01.618Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T05:52:01.820Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-16T05:52:01.831Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-16T05:52:01.843Z] Avoid second fetch [2026-04-16T05:52:01.843Z] Checking out Revision 73ed6275b1ab90dc4db382514cbc5070a56e2e90 (main) [2026-04-16T05:52:01.843Z] > git config core.sparsecheckout # timeout=10 [2026-04-16T05:52:01.850Z] > git checkout -f 73ed6275b1ab90dc4db382514cbc5070a56e2e90 # timeout=10 [2026-04-16T05:52:01.884Z] Commit message: "Revert "New getUpNodesByLabel() call"" [2026-04-16T05:52:01.936Z] [Pipeline] Start of Pipeline [2026-04-16T05:52:02.032Z] [Pipeline] node [2026-04-16T05:52:17.032Z] Still waiting to schedule task [2026-04-16T05:52:17.033Z] Waiting for next available executor on β€˜anvil-bm’ [2026-04-16T07:48:04.487Z] Running on anvil-ci-bm-phy03 in /srv/workspace/anvil/anvil-build-functional-testing [2026-04-16T07:48:04.487Z] [Pipeline] { [2026-04-16T07:48:04.496Z] [Pipeline] stage [2026-04-16T07:48:04.497Z] [Pipeline] { (Prep build env) [2026-04-16T07:48:04.506Z] [Pipeline] script [2026-04-16T07:48:04.508Z] [Pipeline] { [2026-04-16T07:48:04.511Z] [Pipeline] cleanWs [2026-04-16T07:48:04.612Z] [WS-CLEANUP] Deleting project workspace... [2026-04-16T07:48:04.612Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-16T07:48:05.164Z] [WS-CLEANUP] done [2026-04-16T07:48:05.185Z] [Pipeline] lock [2026-04-16T07:48:05.185Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-04-16T07:48:05.185Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-04-16T07:48:05.185Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-04-16T07:48:05.185Z] [Pipeline] { [2026-04-16T07:48:05.187Z] [Pipeline] } [2026-04-16T07:48:05.188Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-04-16T07:48:05.190Z] [Pipeline] // lock [2026-04-16T07:48:05.194Z] [Pipeline] } [2026-04-16T07:48:05.196Z] [Pipeline] // script [2026-04-16T07:48:05.198Z] [Pipeline] } [2026-04-16T07:48:05.200Z] [Pipeline] // stage [2026-04-16T07:48:05.202Z] [Pipeline] stage [2026-04-16T07:48:05.203Z] [Pipeline] { (Run functional testing) [2026-04-16T07:48:05.209Z] [Pipeline] script [2026-04-16T07:48:05.210Z] [Pipeline] { [2026-04-16T07:48:05.231Z] [Pipeline] echo [2026-04-16T07:48:05.232Z] runWithTimeout 360 [2026-04-16T07:48:05.232Z] [Pipeline] timeout [2026-04-16T07:48:05.232Z] Timeout set to expire in 6 hr 0 min [2026-04-16T07:48:05.232Z] [Pipeline] { [2026-04-16T07:48:05.237Z] [Pipeline] sh [2026-04-16T07:48:05.824Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 enterprise pr/1000 all [2026-04-16T07:48:05.825Z] CITBRANCH: main [2026-04-16T07:48:05.825Z] CITHOME: /root/ci-tools [2026-04-16T07:48:05.825Z] CITCMD: ci-setup-anvil-bm-vm [2026-04-16T07:48:05.825Z] CITCMDARGS: rhel-9 enterprise pr/1000 all [2026-04-16T07:48:05.825Z] Genering ssh keys and ssh config [2026-04-16T07:48:06.196Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:48:06.196Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:48:06.196Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:48:06.196Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:48:06.196Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:48:06.196Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-04-16T07:51:16.064Z] Destroying VMs (if any) [2026-04-16T07:51:16.064Z] Warning: Permanently added 'an-striker01' (ED25519) to the list of known hosts. [2026-04-16T07:51:16.064Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-04-16T07:51:16.064Z] This might take a few seconds. [2026-04-16T07:51:16.064Z] [2026-04-16T07:51:16.064Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T07:51:16.064Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T07:51:16.064Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T07:51:16.064Z] [2026-04-16T07:51:16.064Z] Manage your connected systems: https://red.ht/connector [2026-04-16T07:51:16.064Z] [2026-04-16T07:51:16.064Z] The following errors were encountered during disconnect: [2026-04-16T07:51:16.064Z] [2026-04-16T07:51:16.064Z] TYPE STEP ERROR [2026-04-16T07:51:16.064Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T07:51:16.064Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T07:51:19.271Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T07:51:19.271Z] [2026-04-16T07:51:19.271Z] 1. Register with subscription-manager [2026-04-16T07:51:19.271Z] # subscription-manager register [2026-04-16T07:51:19.271Z] [2026-04-16T07:51:19.271Z] 2. Register with insights-client [2026-04-16T07:51:19.271Z] # insights-client --register [2026-04-16T07:51:20.062Z] This system is currently not registered. [2026-04-16T07:51:22.527Z] Domain 'an-striker01' destroyed [2026-04-16T07:51:22.527Z] [2026-04-16T07:51:22.527Z] Domain 'an-striker01' has been undefined [2026-04-16T07:51:22.527Z] [2026-04-16T07:51:22.527Z] Warning: Permanently added 'an-striker02' (ED25519) to the list of known hosts. [2026-04-16T07:51:22.527Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-04-16T07:51:22.527Z] This might take a few seconds. [2026-04-16T07:51:22.527Z] [2026-04-16T07:51:22.527Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T07:51:27.615Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T07:51:27.615Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T07:51:27.615Z] [2026-04-16T07:51:27.615Z] Manage your connected systems: https://red.ht/connector [2026-04-16T07:51:27.615Z] [2026-04-16T07:51:27.615Z] The following errors were encountered during disconnect: [2026-04-16T07:51:27.615Z] [2026-04-16T07:51:27.615Z] TYPE STEP ERROR [2026-04-16T07:51:27.615Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T07:51:27.615Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T07:51:32.238Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T07:51:32.238Z] [2026-04-16T07:51:32.238Z] 1. Register with subscription-manager [2026-04-16T07:51:32.238Z] # subscription-manager register [2026-04-16T07:51:32.238Z] [2026-04-16T07:51:32.238Z] 2. Register with insights-client [2026-04-16T07:51:32.238Z] # insights-client --register [2026-04-16T07:51:32.238Z] This system is currently not registered. [2026-04-16T07:51:34.700Z] Domain 'an-striker02' destroyed [2026-04-16T07:51:34.700Z] [2026-04-16T07:51:34.700Z] Domain 'an-striker02' has been undefined [2026-04-16T07:51:34.700Z] [2026-04-16T07:51:34.700Z] Warning: Permanently added 'an-a01n01' (ED25519) to the list of known hosts. [2026-04-16T07:51:35.070Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-04-16T07:51:35.070Z] This might take a few seconds. [2026-04-16T07:51:35.070Z] [2026-04-16T07:51:35.070Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T07:51:39.140Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T07:51:39.140Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T07:51:39.140Z] [2026-04-16T07:51:39.140Z] Manage your connected systems: https://red.ht/connector [2026-04-16T07:51:39.140Z] [2026-04-16T07:51:39.140Z] The following errors were encountered during disconnect: [2026-04-16T07:51:39.140Z] [2026-04-16T07:51:39.140Z] TYPE STEP ERROR [2026-04-16T07:51:39.140Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T07:51:39.140Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T07:51:43.209Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T07:51:43.209Z] [2026-04-16T07:51:43.209Z] 1. Register with subscription-manager [2026-04-16T07:51:43.209Z] # subscription-manager register [2026-04-16T07:51:43.209Z] [2026-04-16T07:51:43.209Z] 2. Register with insights-client [2026-04-16T07:51:43.209Z] # insights-client --register [2026-04-16T07:51:43.580Z] This system is currently not registered. [2026-04-16T07:51:46.790Z] Domain 'an-a01n01' destroyed [2026-04-16T07:51:46.790Z] [2026-04-16T07:51:46.790Z] Domain 'an-a01n01' has been undefined [2026-04-16T07:51:46.790Z] [2026-04-16T07:51:46.790Z] Warning: Permanently added 'an-a01n02' (ED25519) to the list of known hosts. [2026-04-16T07:51:46.790Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-04-16T07:51:46.790Z] This might take a few seconds. [2026-04-16T07:51:46.790Z] [2026-04-16T07:51:46.790Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T07:51:51.125Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T07:51:51.125Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T07:51:51.125Z] [2026-04-16T07:51:51.125Z] Manage your connected systems: https://red.ht/connector [2026-04-16T07:51:51.125Z] [2026-04-16T07:51:51.126Z] The following errors were encountered during disconnect: [2026-04-16T07:51:51.126Z] [2026-04-16T07:51:51.126Z] TYPE STEP ERROR [2026-04-16T07:51:51.126Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T07:51:51.126Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T07:51:54.330Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T07:51:54.330Z] [2026-04-16T07:51:54.330Z] 1. Register with subscription-manager [2026-04-16T07:51:54.330Z] # subscription-manager register [2026-04-16T07:51:54.330Z] [2026-04-16T07:51:54.330Z] 2. Register with insights-client [2026-04-16T07:51:54.330Z] # insights-client --register [2026-04-16T07:51:55.120Z] This system is currently not registered. [2026-04-16T07:51:57.584Z] Domain 'an-a01n02' destroyed [2026-04-16T07:51:57.584Z] [2026-04-16T07:51:57.584Z] Domain 'an-a01n02' has been undefined [2026-04-16T07:51:57.584Z] [2026-04-16T07:51:57.584Z] Warning: Permanently added 'an-a01dr01' (ED25519) to the list of known hosts. [2026-04-16T07:51:58.375Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-04-16T07:51:58.375Z] This might take a few seconds. [2026-04-16T07:51:58.375Z] [2026-04-16T07:51:58.375Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T07:52:02.446Z] ! Cannot disconnect from Red Hat Insights [2026-04-16T07:52:02.446Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-16T07:52:02.446Z] [2026-04-16T07:52:02.446Z] Manage your connected systems: https://red.ht/connector [2026-04-16T07:52:02.446Z] [2026-04-16T07:52:02.446Z] The following errors were encountered during disconnect: [2026-04-16T07:52:02.446Z] [2026-04-16T07:52:02.446Z] TYPE STEP ERROR [2026-04-16T07:52:02.446Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-16T07:52:02.446Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-16T07:52:05.652Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T07:52:05.652Z] [2026-04-16T07:52:05.652Z] 1. Register with subscription-manager [2026-04-16T07:52:05.652Z] # subscription-manager register [2026-04-16T07:52:05.652Z] [2026-04-16T07:52:05.652Z] 2. Register with insights-client [2026-04-16T07:52:05.652Z] # insights-client --register [2026-04-16T07:52:06.443Z] This system is currently not registered. [2026-04-16T07:52:09.197Z] Domain 'an-a01dr01' destroyed [2026-04-16T07:52:09.197Z] [2026-04-16T07:52:09.197Z] Domain 'an-a01dr01' has been undefined [2026-04-16T07:52:09.197Z] [2026-04-16T07:52:09.197Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-16T07:52:09.197Z] Genering ssh keys and ssh config [2026-04-16T07:52:09.197Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:52:09.197Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:52:09.566Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:52:09.566Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:52:09.566Z] # github.com:22 SSH-2.0-3992d52 [2026-04-16T07:52:09.566Z] Cache already updated [2026-04-16T07:52:09.566Z] Creating nodes disks [2026-04-16T07:52:17.249Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-16T07:52:17.249Z] Generating an-test-deployX kickstart files/isos [2026-04-16T07:52:17.249Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-04-16T07:52:17.249Z] Media current: stdio file, overwriteable [2026-04-16T07:52:17.249Z] Media status : is blank [2026-04-16T07:52:17.249Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T07:52:17.249Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] ISO image produced: 184 sectors [2026-04-16T07:52:17.249Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T07:52:17.249Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-04-16T07:52:17.249Z] Media current: stdio file, overwriteable [2026-04-16T07:52:17.249Z] Media status : is blank [2026-04-16T07:52:17.249Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T07:52:17.249Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] ISO image produced: 184 sectors [2026-04-16T07:52:17.249Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T07:52:17.249Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-04-16T07:52:17.249Z] Media current: stdio file, overwriteable [2026-04-16T07:52:17.249Z] Media status : is blank [2026-04-16T07:52:17.249Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T07:52:17.249Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] ISO image produced: 184 sectors [2026-04-16T07:52:17.249Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T07:52:17.249Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-04-16T07:52:17.249Z] Media current: stdio file, overwriteable [2026-04-16T07:52:17.249Z] Media status : is blank [2026-04-16T07:52:17.249Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T07:52:17.249Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] ISO image produced: 184 sectors [2026-04-16T07:52:17.249Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T07:52:17.249Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-04-16T07:52:17.249Z] Media current: stdio file, overwriteable [2026-04-16T07:52:17.249Z] Media status : is blank [2026-04-16T07:52:17.249Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-16T07:52:17.249Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-16T07:52:17.249Z] ISO image produced: 184 sectors [2026-04-16T07:52:17.249Z] Written to medium : 184 sectors at LBA 0 [2026-04-16T07:52:17.249Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-04-16T07:52:17.249Z] [2026-04-16T07:52:17.249Z] Generating cloud-init configs [2026-04-16T07:52:17.249Z] Generating cloud-init metadata and seed images [2026-04-16T07:52:17.249Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T07:52:17.249Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T07:52:17.619Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T07:52:17.619Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T07:52:17.619Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-16T07:52:17.619Z] Creating VMs [2026-04-16T07:52:19.448Z] [2026-04-16T07:52:19.448Z] Starting install... [2026-04-16T07:52:19.448Z] [2026-04-16T07:52:19.448Z] Creating domain... | 00:00 [2026-04-16T07:52:19.448Z] Domain creation completed. [2026-04-16T07:52:20.719Z] [2026-04-16T07:52:20.719Z] Starting install... [2026-04-16T07:52:20.719Z] [2026-04-16T07:52:20.719Z] Creating domain... | 00:00 [2026-04-16T07:52:20.719Z] Domain creation completed. [2026-04-16T07:52:22.543Z] [2026-04-16T07:52:22.543Z] Starting install... [2026-04-16T07:52:22.543Z] [2026-04-16T07:52:22.543Z] Creating domain... | 00:00 [2026-04-16T07:52:22.543Z] Domain creation completed. [2026-04-16T07:52:24.369Z] [2026-04-16T07:52:24.369Z] Starting install... [2026-04-16T07:52:24.369Z] [2026-04-16T07:52:24.369Z] Creating domain... | 00:00 [2026-04-16T07:52:24.369Z] Domain creation completed. [2026-04-16T07:52:25.641Z] [2026-04-16T07:52:25.641Z] Starting install... [2026-04-16T07:52:25.641Z] [2026-04-16T07:52:25.641Z] Creating domain... | 00:00 [2026-04-16T07:52:25.641Z] Domain creation completed. [2026-04-16T07:52:25.641Z] Waiting for nodes and cloud-init to complete full init [2026-04-16T07:52:29.112Z] cloud-init status node an-striker01: [2026-04-16T07:52:32.320Z] cloud-init status node an-striker02: [2026-04-16T07:52:35.531Z] cloud-init status node an-a01n01: [2026-04-16T07:52:38.001Z] cloud-init status node an-a01n02: [2026-04-16T07:52:41.212Z] cloud-init status node an-a01dr01: [2026-04-16T07:52:41.212Z] Thu Apr 16 03:52:40 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:52:52.775Z] cloud-init status node an-striker01: running [2026-04-16T07:52:53.568Z] cloud-init status node an-striker02: running [2026-04-16T07:52:54.844Z] cloud-init status node an-a01n01: running [2026-04-16T07:52:56.119Z] cloud-init status node an-a01n02: running [2026-04-16T07:52:57.394Z] cloud-init status node an-a01dr01: running [2026-04-16T07:52:57.394Z] Thu Apr 16 03:52:57 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:53:08.757Z] cloud-init status node an-striker01: running [2026-04-16T07:53:08.757Z] cloud-init status node an-striker02: running [2026-04-16T07:53:10.031Z] cloud-init status node an-a01n01: running [2026-04-16T07:53:10.824Z] cloud-init status node an-a01n02: running [2026-04-16T07:53:11.616Z] cloud-init status node an-a01dr01: running [2026-04-16T07:53:11.616Z] Thu Apr 16 03:53:11 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:53:23.183Z] cloud-init status node an-striker01: running [2026-04-16T07:53:23.183Z] cloud-init status node an-striker02: running [2026-04-16T07:53:24.458Z] cloud-init status node an-a01n01: running [2026-04-16T07:53:25.250Z] cloud-init status node an-a01n02: running [2026-04-16T07:53:26.043Z] cloud-init status node an-a01dr01: running [2026-04-16T07:53:26.043Z] Thu Apr 16 03:53:25 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:53:39.722Z] cloud-init status node an-striker01: running [2026-04-16T07:53:39.722Z] cloud-init status node an-striker02: running [2026-04-16T07:53:40.516Z] cloud-init status node an-a01n01: running [2026-04-16T07:53:41.792Z] cloud-init status node an-a01n02: running [2026-04-16T07:53:42.795Z] cloud-init status node an-a01dr01: running [2026-04-16T07:53:42.795Z] Thu Apr 16 03:53:42 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:53:54.130Z] cloud-init status node an-striker01: running [2026-04-16T07:53:54.505Z] cloud-init status node an-striker02: running [2026-04-16T07:53:55.302Z] cloud-init status node an-a01n01: running [2026-04-16T07:54:01.678Z] cloud-init status node an-a01n02: running [2026-04-16T07:54:02.477Z] cloud-init status node an-a01dr01: running [2026-04-16T07:54:02.477Z] Thu Apr 16 03:54:02 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:54:13.867Z] cloud-init status node an-striker01: running [2026-04-16T07:54:14.245Z] cloud-init status node an-striker02: running [2026-04-16T07:54:15.535Z] cloud-init status node an-a01n01: running [2026-04-16T07:54:16.337Z] cloud-init status node an-a01n02: running [2026-04-16T07:54:17.140Z] cloud-init status node an-a01dr01: running [2026-04-16T07:54:17.140Z] Thu Apr 16 03:54:16 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:54:28.521Z] cloud-init status node an-striker01: running [2026-04-16T07:54:29.800Z] cloud-init status node an-striker02: running [2026-04-16T07:54:30.483Z] cloud-init status node an-a01n01: running [2026-04-16T07:54:31.757Z] cloud-init status node an-a01n02: running [2026-04-16T07:54:32.551Z] cloud-init status node an-a01dr01: running [2026-04-16T07:54:32.551Z] Thu Apr 16 03:54:32 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:54:44.789Z] cloud-init status node an-striker01: running [2026-04-16T07:54:44.789Z] cloud-init status node an-striker02: running [2026-04-16T07:54:45.159Z] cloud-init status node an-a01n01: running [2026-04-16T07:54:45.951Z] cloud-init status node an-a01n02: running [2026-04-16T07:54:46.743Z] cloud-init status node an-a01dr01: running [2026-04-16T07:54:46.743Z] Thu Apr 16 03:54:46 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:54:58.076Z] cloud-init status node an-striker01: running [2026-04-16T07:54:58.868Z] cloud-init status node an-striker02: running [2026-04-16T07:54:59.661Z] cloud-init status node an-a01n01: running [2026-04-16T07:55:00.454Z] cloud-init status node an-a01n02: running [2026-04-16T07:55:01.247Z] cloud-init status node an-a01dr01: running [2026-04-16T07:55:01.247Z] Thu Apr 16 03:55:01 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:55:12.629Z] cloud-init status node an-striker01: running [2026-04-16T07:55:13.000Z] cloud-init status node an-striker02: running [2026-04-16T07:55:13.793Z] cloud-init status node an-a01n01: running [2026-04-16T07:55:15.069Z] cloud-init status node an-a01n02: running [2026-04-16T07:55:15.863Z] cloud-init status node an-a01dr01: running [2026-04-16T07:55:15.863Z] Thu Apr 16 03:55:15 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:55:27.194Z] cloud-init status node an-striker01: running [2026-04-16T07:55:27.565Z] cloud-init status node an-striker02: running [2026-04-16T07:55:28.362Z] cloud-init status node an-a01n01: running [2026-04-16T07:55:29.508Z] cloud-init status node an-a01n02: running [2026-04-16T07:55:30.300Z] cloud-init status node an-a01dr01: running [2026-04-16T07:55:30.300Z] Thu Apr 16 03:55:30 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:55:41.640Z] cloud-init status node an-striker01: running [2026-04-16T07:55:42.433Z] cloud-init status node an-striker02: running [2026-04-16T07:55:43.229Z] cloud-init status node an-a01n01: running [2026-04-16T07:55:44.022Z] cloud-init status node an-a01n02: running [2026-04-16T07:55:44.816Z] cloud-init status node an-a01dr01: running [2026-04-16T07:55:44.816Z] Thu Apr 16 03:55:44 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:55:56.147Z] cloud-init status node an-striker01: running [2026-04-16T07:55:56.971Z] cloud-init status node an-striker02: running [2026-04-16T07:55:57.771Z] cloud-init status node an-a01n01: running [2026-04-16T07:55:58.580Z] cloud-init status node an-a01n02: running [2026-04-16T07:55:59.375Z] cloud-init status node an-a01dr01: running [2026-04-16T07:55:59.375Z] Thu Apr 16 03:55:59 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:56:10.810Z] cloud-init status node an-striker01: running [2026-04-16T07:56:11.182Z] cloud-init status node an-striker02: running [2026-04-16T07:56:12.458Z] cloud-init status node an-a01n01: running [2026-04-16T07:56:13.251Z] cloud-init status node an-a01n02: running [2026-04-16T07:56:14.044Z] cloud-init status node an-a01dr01: running [2026-04-16T07:56:14.044Z] Thu Apr 16 03:56:13 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:56:25.380Z] cloud-init status node an-striker01: running [2026-04-16T07:56:26.175Z] cloud-init status node an-striker02: running [2026-04-16T07:56:27.451Z] cloud-init status node an-a01n01: running [2026-04-16T07:56:28.253Z] cloud-init status node an-a01n02: running [2026-04-16T07:56:29.047Z] cloud-init status node an-a01dr01: running [2026-04-16T07:56:29.047Z] Thu Apr 16 03:56:28 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:56:40.378Z] cloud-init status node an-striker01: running [2026-04-16T07:56:41.176Z] cloud-init status node an-striker02: running [2026-04-16T07:56:41.970Z] cloud-init status node an-a01n01: running [2026-04-16T07:56:42.763Z] cloud-init status node an-a01n02: running [2026-04-16T07:56:44.040Z] cloud-init status node an-a01dr01: running [2026-04-16T07:56:44.040Z] Thu Apr 16 03:56:43 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:56:55.875Z] cloud-init status node an-striker01: running [2026-04-16T07:56:55.875Z] cloud-init status node an-striker02: running [2026-04-16T07:56:56.670Z] cloud-init status node an-a01n01: running [2026-04-16T07:56:57.464Z] cloud-init status node an-a01n02: running [2026-04-16T07:56:58.738Z] cloud-init status node an-a01dr01: running [2026-04-16T07:56:58.738Z] Thu Apr 16 03:56:58 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:57:10.076Z] cloud-init status node an-striker01: running [2026-04-16T07:57:10.449Z] cloud-init status node an-striker02: running [2026-04-16T07:57:11.246Z] cloud-init status node an-a01n01: running [2026-04-16T07:57:12.139Z] cloud-init status node an-a01n02: running [2026-04-16T07:57:14.359Z] cloud-init status node an-a01dr01: running [2026-04-16T07:57:14.359Z] Thu Apr 16 03:57:13 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:57:25.765Z] cloud-init status node an-striker01: running [2026-04-16T07:57:26.558Z] cloud-init status node an-striker02: running [2026-04-16T07:57:27.349Z] cloud-init status node an-a01n01: running [2026-04-16T07:57:28.236Z] cloud-init status node an-a01n02: running [2026-04-16T07:57:29.512Z] cloud-init status node an-a01dr01: running [2026-04-16T07:57:29.512Z] Thu Apr 16 03:57:29 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:57:40.848Z] cloud-init status node an-striker01: running [2026-04-16T07:57:41.219Z] cloud-init status node an-striker02: running [2026-04-16T07:57:42.243Z] cloud-init status node an-a01n01: running [2026-04-16T07:57:43.037Z] cloud-init status node an-a01n02: running [2026-04-16T07:57:44.315Z] cloud-init status node an-a01dr01: running [2026-04-16T07:57:44.315Z] Thu Apr 16 03:57:43 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:57:55.657Z] cloud-init status node an-striker01: running [2026-04-16T07:57:56.028Z] cloud-init status node an-striker02: running [2026-04-16T07:57:56.821Z] cloud-init status node an-a01n01: running [2026-04-16T07:57:58.096Z] cloud-init status node an-a01n02: running [2026-04-16T07:57:58.888Z] cloud-init status node an-a01dr01: running [2026-04-16T07:57:58.888Z] Thu Apr 16 03:57:58 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:58:10.760Z] cloud-init status node an-striker01: running [2026-04-16T07:58:10.760Z] cloud-init status node an-striker02: running [2026-04-16T07:58:12.035Z] cloud-init status node an-a01n01: running [2026-04-16T07:58:12.828Z] cloud-init status node an-a01n02: running [2026-04-16T07:58:13.623Z] cloud-init status node an-a01dr01: running [2026-04-16T07:58:13.623Z] Thu Apr 16 03:58:13 EDT 2026 actual: 0 expected: 5 [2026-04-16T07:58:25.022Z] cloud-init status node an-striker01: running [2026-04-16T07:58:26.298Z] cloud-init status node an-striker02: running [2026-04-16T07:58:27.579Z] cloud-init status node an-a01n01: done [2026-04-16T07:58:28.857Z] cloud-init status node an-a01n02: running [2026-04-16T07:58:32.480Z] cloud-init status node an-a01dr01: running [2026-04-16T07:58:32.480Z] Thu Apr 16 03:58:31 EDT 2026 actual: 1 expected: 5 [2026-04-16T07:58:44.115Z] cloud-init status node an-striker01: running [2026-04-16T07:58:44.909Z] cloud-init status node an-striker02: running [2026-04-16T07:58:46.183Z] cloud-init status node an-a01n01: done [2026-04-16T07:58:47.468Z] cloud-init status node an-a01n02: done [2026-04-16T07:58:48.745Z] cloud-init status node an-a01dr01: done [2026-04-16T07:58:48.745Z] Thu Apr 16 03:58:48 EDT 2026 actual: 3 expected: 5 [2026-04-16T07:59:02.426Z] cloud-init status node an-striker01: running [2026-04-16T07:59:02.426Z] cloud-init status node an-striker02: done [2026-04-16T07:59:02.858Z] cloud-init status node an-a01n01: done [2026-04-16T07:59:03.658Z] cloud-init status node an-a01n02: done [2026-04-16T07:59:04.940Z] cloud-init status node an-a01dr01: done [2026-04-16T07:59:04.940Z] Thu Apr 16 03:59:04 EDT 2026 actual: 4 expected: 5 [2026-04-16T07:59:16.324Z] cloud-init status node an-striker01: done [2026-04-16T07:59:17.598Z] cloud-init status node an-striker02: done [2026-04-16T07:59:18.391Z] cloud-init status node an-a01n01: done [2026-04-16T07:59:19.675Z] cloud-init status node an-a01n02: done [2026-04-16T07:59:20.476Z] cloud-init status node an-a01dr01: done [2026-04-16T07:59:20.476Z] Thu Apr 16 03:59:20 EDT 2026 actual: 5 expected: 5 [2026-04-16T07:59:20.476Z] Building simengine model [2026-04-16T07:59:35.064Z] Generating anvil striker auto config [2026-04-16T07:59:35.864Z] Cloning into 'anvil-test-suite'... [2026-04-16T07:59:36.667Z] generated ansible config: [2026-04-16T07:59:36.667Z] ========================= [2026-04-16T07:59:36.667Z] [defaults] [2026-04-16T07:59:36.667Z] inventory = ansible-inventory [2026-04-16T07:59:36.667Z] forks = 500 [2026-04-16T07:59:36.667Z] any_errors_fatal = true [2026-04-16T07:59:36.667Z] display_args_to_stdout = true [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:36.667Z] [ssh_connection] [2026-04-16T07:59:36.667Z] retries = 20 [2026-04-16T07:59:36.667Z] ========================= [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:36.667Z] generated ansible inventory: [2026-04-16T07:59:36.667Z] ============================ [2026-04-16T07:59:36.667Z] [strikers] [2026-04-16T07:59:36.667Z] an-striker01 [2026-04-16T07:59:36.667Z] an-striker02 [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:36.667Z] [nodes] [2026-04-16T07:59:36.667Z] an-a01n01 [2026-04-16T07:59:36.667Z] an-a01n02 [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:36.667Z] [drnodes] [2026-04-16T07:59:36.667Z] an-a01dr01 [2026-04-16T07:59:36.667Z] ============================ [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:36.667Z] Checking connectivity to nodes [2026-04-16T07:59:36.667Z] [2026-04-16T07:59:39.197Z] an-a01n02 | SUCCESS => { [2026-04-16T07:59:39.197Z] "ansible_facts": { [2026-04-16T07:59:39.197Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T07:59:39.197Z] }, [2026-04-16T07:59:39.197Z] "changed": false, [2026-04-16T07:59:39.197Z] "ping": "pong" [2026-04-16T07:59:39.197Z] } [2026-04-16T07:59:39.197Z] an-a01n01 | SUCCESS => { [2026-04-16T07:59:39.197Z] "ansible_facts": { [2026-04-16T07:59:39.197Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T07:59:39.197Z] }, [2026-04-16T07:59:39.197Z] "changed": false, [2026-04-16T07:59:39.197Z] "ping": "pong" [2026-04-16T07:59:39.197Z] } [2026-04-16T07:59:39.197Z] an-a01dr01 | SUCCESS => { [2026-04-16T07:59:39.197Z] "ansible_facts": { [2026-04-16T07:59:39.197Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T07:59:39.197Z] }, [2026-04-16T07:59:39.197Z] "changed": false, [2026-04-16T07:59:39.197Z] "ping": "pong" [2026-04-16T07:59:39.197Z] } [2026-04-16T07:59:39.197Z] an-striker01 | SUCCESS => { [2026-04-16T07:59:39.197Z] "ansible_facts": { [2026-04-16T07:59:39.197Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T07:59:39.197Z] }, [2026-04-16T07:59:39.197Z] "changed": false, [2026-04-16T07:59:39.197Z] "ping": "pong" [2026-04-16T07:59:39.197Z] } [2026-04-16T07:59:39.197Z] an-striker02 | SUCCESS => { [2026-04-16T07:59:39.197Z] "ansible_facts": { [2026-04-16T07:59:39.197Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-16T07:59:39.197Z] }, [2026-04-16T07:59:39.197Z] "changed": false, [2026-04-16T07:59:39.197Z] "ping": "pong" [2026-04-16T07:59:39.197Z] } [2026-04-16T07:59:39.197Z] [2026-04-16T07:59:39.580Z] [2026-04-16T07:59:39.580Z] PLAY [Setup all nodes] ********************************************************* [2026-04-16T07:59:39.580Z] [2026-04-16T07:59:39.580Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:59:42.081Z] ok: [an-a01n01] [2026-04-16T07:59:42.081Z] ok: [an-a01n02] [2026-04-16T07:59:42.081Z] ok: [an-a01dr01] [2026-04-16T07:59:42.081Z] ok: [an-striker02] [2026-04-16T07:59:42.081Z] ok: [an-striker01] [2026-04-16T07:59:42.081Z] [2026-04-16T07:59:42.081Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-04-16T07:59:42.920Z] changed: [an-a01n02] [2026-04-16T07:59:42.920Z] changed: [an-a01n01] [2026-04-16T07:59:42.920Z] changed: [an-a01dr01] [2026-04-16T07:59:42.920Z] changed: [an-striker01] [2026-04-16T07:59:42.920Z] changed: [an-striker02] [2026-04-16T07:59:42.920Z] [2026-04-16T07:59:42.920Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-04-16T07:59:43.755Z] ok: [an-a01n01] [2026-04-16T07:59:43.755Z] ok: [an-a01n02] [2026-04-16T07:59:43.755Z] ok: [an-a01dr01] [2026-04-16T07:59:43.755Z] ok: [an-striker01] [2026-04-16T07:59:43.755Z] ok: [an-striker02] [2026-04-16T07:59:43.755Z] [2026-04-16T07:59:43.755Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-04-16T07:59:45.032Z] changed: [an-a01n01] [2026-04-16T07:59:45.032Z] changed: [an-a01dr01] [2026-04-16T07:59:45.032Z] changed: [an-a01n02] [2026-04-16T07:59:45.032Z] changed: [an-striker02] [2026-04-16T07:59:45.032Z] changed: [an-striker01] [2026-04-16T07:59:45.032Z] [2026-04-16T07:59:45.032Z] PLAY [setup strikers] ********************************************************** [2026-04-16T07:59:45.032Z] [2026-04-16T07:59:45.032Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:59:46.310Z] ok: [an-striker02] [2026-04-16T07:59:46.310Z] ok: [an-striker01] [2026-04-16T07:59:46.310Z] [2026-04-16T07:59:46.310Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-04-16T07:59:48.442Z] changed: [an-striker02] [2026-04-16T07:59:48.442Z] changed: [an-striker01] [2026-04-16T07:59:48.442Z] [2026-04-16T07:59:48.442Z] PLAY [setup lvm storage] ******************************************************* [2026-04-16T07:59:48.442Z] [2026-04-16T07:59:48.442Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:59:50.483Z] ok: [an-a01n02] [2026-04-16T07:59:50.483Z] ok: [an-a01n01] [2026-04-16T07:59:50.483Z] ok: [an-a01dr01] [2026-04-16T07:59:50.483Z] [2026-04-16T07:59:50.483Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-04-16T07:59:50.483Z] ] **************** [2026-04-16T07:59:50.483Z] ok: [an-a01n02] [2026-04-16T07:59:50.483Z] ok: [an-a01n01] [2026-04-16T07:59:50.483Z] ok: [an-a01dr01] [2026-04-16T07:59:50.483Z] [2026-04-16T07:59:50.483Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-04-16T07:59:50.483Z] skipping: [an-a01n01] [2026-04-16T07:59:50.483Z] [2026-04-16T07:59:50.483Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-04-16T07:59:51.763Z] changed: [an-a01n02] [2026-04-16T07:59:51.763Z] changed: [an-a01dr01] [2026-04-16T07:59:51.763Z] changed: [an-a01n01] [2026-04-16T07:59:51.763Z] [2026-04-16T07:59:51.763Z] PLAY [deploy anvil] ************************************************************ [2026-04-16T07:59:51.763Z] [2026-04-16T07:59:51.763Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T07:59:53.590Z] ok: [an-striker01] [2026-04-16T07:59:53.590Z] ok: [an-striker02] [2026-04-16T07:59:53.590Z] [2026-04-16T07:59:53.590Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-04-16T07:59:53.962Z] ok: [an-striker01] [2026-04-16T07:59:53.962Z] ok: [an-striker02] [2026-04-16T07:59:53.962Z] [2026-04-16T07:59:53.962Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-04-16T07:59:55.913Z] changed: [an-striker01] [2026-04-16T07:59:55.913Z] changed: [an-striker02] [2026-04-16T07:59:55.913Z] [2026-04-16T07:59:55.913Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-04-16T07:59:55.913Z] ] *** [2026-04-16T07:59:58.380Z] changed: [an-striker01] [2026-04-16T07:59:58.380Z] changed: [an-striker02] [2026-04-16T07:59:58.380Z] [2026-04-16T07:59:58.380Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T07:59:58.380Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T07:59:58.380Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-04-16T07:59:58.380Z] ] *** [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-16T08:05:40.814Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-16T08:05:40.815Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-16T08:05:40.815Z] ok: [an-striker02 -> localhost] [2026-04-16T08:05:40.815Z] ok: [an-striker01 -> localhost] [2026-04-16T08:05:40.815Z] [2026-04-16T08:05:40.815Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-04-16T08:05:40.815Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-04-16T08:05:40.815Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-04-16T08:05:40.815Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-04-16T08:05:40.815Z] echo "DB has completed sync" [2026-04-16T08:05:40.815Z] exit 0 [2026-04-16T08:05:40.815Z] fi [2026-04-16T08:05:40.815Z] if [ "$dbcon" != 2 ]; then [2026-04-16T08:05:40.815Z] echo "Detected $dbcon db connections (expected 2)" [2026-04-16T08:05:40.815Z] exit 1 [2026-04-16T08:05:40.815Z] fi [2026-04-16T08:05:40.815Z] ] *** [2026-04-16T08:06:04.768Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-04-16T08:06:04.768Z] ok: [an-striker01] [2026-04-16T08:06:04.768Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-04-16T08:06:04.768Z] ok: [an-striker02] [2026-04-16T08:06:04.768Z] [2026-04-16T08:06:04.768Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T08:06:04.768Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T08:06:04.768Z] --username=postgres anvil -t \ [2026-04-16T08:06:04.768Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-04-16T08:06:04.768Z] grep altee | \ [2026-04-16T08:06:04.768Z] grep 1$ | \ [2026-04-16T08:06:04.768Z] wc -l [2026-04-16T08:06:04.768Z] ] *** [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-16T08:11:47.557Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-16T08:11:47.558Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-16T08:11:47.558Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-16T08:11:47.558Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-04-16T08:15:28.253Z] ok: [an-striker01 -> localhost] [2026-04-16T08:15:28.253Z] ok: [an-striker02 -> localhost] [2026-04-16T08:15:28.253Z] [2026-04-16T08:15:28.253Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T08:15:28.253Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T08:15:28.253Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-04-16T08:15:28.253Z] grep altee | \ [2026-04-16T08:15:28.253Z] wc -l [2026-04-16T08:15:28.253Z] ] *** [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-04-16T08:15:28.253Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-04-16T08:15:28.253Z] ok: [an-striker01 -> localhost] [2026-04-16T08:15:28.253Z] ok: [an-striker02 -> localhost] [2026-04-16T08:15:28.253Z] [2026-04-16T08:15:28.253Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T08:15:28.253Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T08:15:28.253Z] --username=postgres anvil -t \ [2026-04-16T08:15:28.253Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-04-16T08:15:28.253Z] grep altee | \ [2026-04-16T08:15:28.253Z] grep 1$ | \ [2026-04-16T08:15:28.253Z] wc -l [2026-04-16T08:15:28.253Z] ] *** [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-16T08:17:37.653Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-16T08:17:37.653Z] ok: [an-striker01 -> localhost] [2026-04-16T08:17:37.653Z] ok: [an-striker02 -> localhost] [2026-04-16T08:17:37.653Z] [2026-04-16T08:17:37.653Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T08:17:37.653Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T08:17:37.653Z] --username=postgres anvil -t \ [2026-04-16T08:17:37.653Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-04-16T08:17:37.653Z] grep altee | \ [2026-04-16T08:17:37.653Z] grep 1$ | \ [2026-04-16T08:17:37.653Z] wc -l [2026-04-16T08:17:37.653Z] ] *** [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left). [2026-04-16T08:18:06.769Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left). [2026-04-16T08:18:06.769Z] ok: [an-striker01 -> localhost] [2026-04-16T08:18:06.769Z] ok: [an-striker02 -> localhost] [2026-04-16T08:18:06.769Z] [2026-04-16T08:18:06.769Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-04-16T08:18:06.769Z] psql --host="{{ inventory_hostname }}" \ [2026-04-16T08:18:06.769Z] --username=postgres anvil -t \ [2026-04-16T08:18:06.769Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-04-16T08:18:06.769Z] grep altee | \ [2026-04-16T08:18:06.769Z] grep 1$ | \ [2026-04-16T08:18:06.769Z] wc -l [2026-04-16T08:18:06.769Z] ] *** [2026-04-16T08:18:06.769Z] ok: [an-striker01 -> localhost] [2026-04-16T08:18:06.769Z] ok: [an-striker02 -> localhost] [2026-04-16T08:18:06.769Z] [2026-04-16T08:18:06.769Z] PLAY [check network status after deployment] *********************************** [2026-04-16T08:18:06.769Z] [2026-04-16T08:18:06.769Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:18:06.769Z] ok: [an-striker01] [2026-04-16T08:18:06.769Z] ok: [an-a01n02] [2026-04-16T08:18:06.769Z] ok: [an-a01dr01] [2026-04-16T08:18:06.769Z] ok: [an-a01n01] [2026-04-16T08:18:06.769Z] ok: [an-striker02] [2026-04-16T08:18:06.770Z] [2026-04-16T08:18:06.770Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-04-16T08:18:06.770Z] if [ "$device" = "" ]; then [2026-04-16T08:18:06.770Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-04-16T08:18:06.770Z] exit 1 [2026-04-16T08:18:06.770Z] fi [2026-04-16T08:18:06.770Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-04-16T08:18:06.770Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-04-16T08:18:06.770Z] exit 1 [2026-04-16T08:18:06.770Z] fi [2026-04-16T08:18:06.770Z] exit 0 [2026-04-16T08:18:06.770Z] ] *** [2026-04-16T08:18:06.770Z] skipping: [an-striker01] => (item=lo) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=ifn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=ifn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=ifn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=bcn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=bcn1_link1) [2026-04-16T08:18:14.462Z] skipping: [an-striker02] => (item=lo) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-04-16T08:18:14.462Z] skipping: [an-a01n01] => (item=lo) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=sn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=ifn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=bcn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=sn1_link2) [2026-04-16T08:18:14.462Z] skipping: [an-a01dr01] => (item=lo) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=sn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=ifn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=bcn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=bcn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=sn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=ifn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=bcn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-striker01] => (item=bcn1_link2) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=sn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-04-16T08:18:14.462Z] skipping: [an-a01n02] => (item=lo) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=bcn1_bridge1) [2026-04-16T08:18:14.462Z] changed: [an-striker02] => (item=ifn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=sn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=ifn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-04-16T08:18:14.462Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-04-16T08:18:14.462Z] changed: [an-a01dr01] => (item=sn1_link1) [2026-04-16T08:18:14.462Z] [2026-04-16T08:18:14.462Z] PLAY [check pacemaker status after deployment] ********************************* [2026-04-16T08:18:14.462Z] [2026-04-16T08:18:14.462Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:18:15.737Z] ok: [an-a01n01] [2026-04-16T08:18:15.737Z] ok: [an-a01n02] [2026-04-16T08:18:15.737Z] [2026-04-16T08:18:15.737Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-04-16T08:18:15.737Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T08:18:15.737Z] [2026-04-16T08:18:15.737Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T08:18:15.737Z] ] *** [2026-04-16T08:18:19.415Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T08:18:19.415Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T08:18:19.415Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T08:18:19.415Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T08:18:19.415Z] [2026-04-16T08:18:19.415Z] PLAY [check drbd kmod] ********************************************************* [2026-04-16T08:18:19.415Z] [2026-04-16T08:18:19.415Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:18:21.242Z] ok: [an-a01dr01] [2026-04-16T08:18:21.242Z] ok: [an-a01n01] [2026-04-16T08:18:21.242Z] ok: [an-a01n02] [2026-04-16T08:18:21.242Z] [2026-04-16T08:18:21.242Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-04-16T08:18:21.242Z] ] *** [2026-04-16T08:18:21.613Z] ok: [an-a01n01] [2026-04-16T08:18:21.613Z] ok: [an-a01n02] [2026-04-16T08:18:21.613Z] ok: [an-a01dr01] [2026-04-16T08:18:21.613Z] [2026-04-16T08:18:21.613Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-04-16T08:18:21.613Z] ] *********************** [2026-04-16T08:18:21.613Z] skipping: [an-a01n01] [2026-04-16T08:18:21.613Z] skipping: [an-a01n02] [2026-04-16T08:18:21.613Z] skipping: [an-a01dr01] [2026-04-16T08:18:21.613Z] [2026-04-16T08:18:21.613Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-04-16T08:18:21.613Z] ] *** [2026-04-16T08:18:21.985Z] skipping: [an-a01n01] [2026-04-16T08:18:21.985Z] skipping: [an-a01n02] [2026-04-16T08:18:21.985Z] skipping: [an-a01dr01] [2026-04-16T08:18:21.985Z] [2026-04-16T08:18:21.985Z] PLAY [Copy iso images to striker01] ******************************************** [2026-04-16T08:18:21.985Z] [2026-04-16T08:18:21.985Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:18:23.814Z] ok: [an-striker01] [2026-04-16T08:18:23.814Z] [2026-04-16T08:18:23.814Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-04-16T08:18:24.606Z] ok: [an-striker01] [2026-04-16T08:18:24.606Z] [2026-04-16T08:18:24.606Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-04-16T08:18:25.398Z] changed: [an-striker01] [2026-04-16T08:18:25.398Z] [2026-04-16T08:18:25.398Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-04-16T08:18:33.122Z] changed: [an-striker01] => (item=deploy1.iso) [2026-04-16T08:18:33.122Z] changed: [an-striker01] => (item=deploy2.iso) [2026-04-16T08:18:33.122Z] changed: [an-striker01] => (item=deploy3.iso) [2026-04-16T08:18:33.122Z] changed: [an-striker01] => (item=deploy4.iso) [2026-04-16T08:18:33.122Z] changed: [an-striker01] => (item=deploy5.iso) [2026-04-16T08:18:33.122Z] [2026-04-16T08:18:33.122Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-04-16T08:18:33.122Z] ] *** [2026-04-16T08:18:33.916Z] changed: [an-striker01] [2026-04-16T08:18:33.916Z] [2026-04-16T08:18:33.916Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-04-16T08:18:34.709Z] changed: [an-striker01] [2026-04-16T08:18:34.709Z] [2026-04-16T08:18:34.709Z] PLAY [Check iso images propagation] ******************************************** [2026-04-16T08:18:34.709Z] [2026-04-16T08:18:34.709Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:18:37.174Z] ok: [an-a01n01] [2026-04-16T08:18:37.174Z] ok: [an-striker01] [2026-04-16T08:18:37.174Z] ok: [an-a01n02] [2026-04-16T08:18:37.174Z] ok: [an-a01dr01] [2026-04-16T08:18:37.174Z] ok: [an-striker02] [2026-04-16T08:18:37.174Z] [2026-04-16T08:18:37.174Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-04-16T08:18:37.174Z] exit 1 [2026-04-16T08:18:37.174Z] else [2026-04-16T08:18:37.174Z] exit 0 [2026-04-16T08:18:37.174Z] fi [2026-04-16T08:18:37.174Z] ] *** [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T08:20:16.699Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-04-16T08:20:16.699Z] ok: [an-a01n02] [2026-04-16T08:20:16.699Z] ok: [an-a01dr01] [2026-04-16T08:20:16.699Z] ok: [an-a01n01] [2026-04-16T08:20:16.699Z] ok: [an-striker01] [2026-04-16T08:20:16.699Z] ok: [an-striker02] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] PLAY [example tests] *********************************************************** [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:20:16.699Z] ok: [an-a01n01] [2026-04-16T08:20:16.699Z] ok: [an-a01n02] [2026-04-16T08:20:16.699Z] ok: [an-a01dr01] [2026-04-16T08:20:16.699Z] ok: [an-striker01] [2026-04-16T08:20:16.699Z] ok: [an-striker02] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-04-16T08:20:16.699Z] ] *** [2026-04-16T08:20:16.699Z] skipping: [an-a01n01] [2026-04-16T08:20:16.699Z] skipping: [an-a01n02] [2026-04-16T08:20:16.699Z] skipping: [an-a01dr01] [2026-04-16T08:20:16.699Z] ok: [an-striker02] [2026-04-16T08:20:16.699Z] ok: [an-striker01] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-04-16T08:20:16.699Z] ] *** [2026-04-16T08:20:16.699Z] skipping: [an-striker01] [2026-04-16T08:20:16.699Z] skipping: [an-striker02] [2026-04-16T08:20:16.699Z] ok: [an-a01n01] [2026-04-16T08:20:16.699Z] ok: [an-a01dr01] [2026-04-16T08:20:16.699Z] ok: [an-a01n02] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [inject failure _raw_params=/usr/bin/false [2026-04-16T08:20:16.699Z] ] ******************************* [2026-04-16T08:20:16.699Z] skipping: [an-striker01] [2026-04-16T08:20:16.699Z] skipping: [an-striker02] [2026-04-16T08:20:16.699Z] skipping: [an-a01n01] [2026-04-16T08:20:16.699Z] skipping: [an-a01n02] [2026-04-16T08:20:16.699Z] skipping: [an-a01dr01] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] PLAY [Record anvil available resources from striker01] ************************* [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:20:16.699Z] ok: [an-striker01] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-04-16T08:20:16.699Z] ] *** [2026-04-16T08:20:16.699Z] ok: [an-striker01] [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] PLAY [check selinux status] **************************************************** [2026-04-16T08:20:16.699Z] [2026-04-16T08:20:16.699Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:20:17.973Z] ok: [an-a01n01] [2026-04-16T08:20:17.973Z] ok: [an-a01n02] [2026-04-16T08:20:17.973Z] ok: [an-a01dr01] [2026-04-16T08:20:17.973Z] ok: [an-striker01] [2026-04-16T08:20:17.973Z] ok: [an-striker02] [2026-04-16T08:20:17.973Z] [2026-04-16T08:20:17.973Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-04-16T08:20:17.973Z] skipping: [an-striker01] [2026-04-16T08:20:17.973Z] skipping: [an-striker02] [2026-04-16T08:20:17.973Z] skipping: [an-a01n01] [2026-04-16T08:20:17.973Z] skipping: [an-a01n02] [2026-04-16T08:20:17.973Z] skipping: [an-a01dr01] [2026-04-16T08:20:17.973Z] [2026-04-16T08:20:17.973Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-04-16T08:20:17.973Z] ok: [an-striker01] => { [2026-04-16T08:20:17.973Z] "ansible_selinux": { [2026-04-16T08:20:17.973Z] "config_mode": "enforcing", [2026-04-16T08:20:17.973Z] "mode": "enforcing", [2026-04-16T08:20:17.973Z] "policyvers": 33, [2026-04-16T08:20:17.973Z] "status": "enabled", [2026-04-16T08:20:17.973Z] "type": "targeted" [2026-04-16T08:20:17.973Z] }, [2026-04-16T08:20:17.973Z] "failed_when_result": false [2026-04-16T08:20:17.973Z] } [2026-04-16T08:20:18.345Z] ok: [an-striker02] => { [2026-04-16T08:20:18.345Z] "ansible_selinux": { [2026-04-16T08:20:18.345Z] "config_mode": "enforcing", [2026-04-16T08:20:18.345Z] "mode": "enforcing", [2026-04-16T08:20:18.345Z] "policyvers": 33, [2026-04-16T08:20:18.345Z] "status": "enabled", [2026-04-16T08:20:18.345Z] "type": "targeted" [2026-04-16T08:20:18.345Z] }, [2026-04-16T08:20:18.345Z] "failed_when_result": false [2026-04-16T08:20:18.345Z] } [2026-04-16T08:20:18.345Z] ok: [an-a01n01] => { [2026-04-16T08:20:18.345Z] "ansible_selinux": { [2026-04-16T08:20:18.345Z] "config_mode": "enforcing", [2026-04-16T08:20:18.345Z] "mode": "enforcing", [2026-04-16T08:20:18.345Z] "policyvers": 33, [2026-04-16T08:20:18.345Z] "status": "enabled", [2026-04-16T08:20:18.345Z] "type": "targeted" [2026-04-16T08:20:18.345Z] }, [2026-04-16T08:20:18.345Z] "failed_when_result": false [2026-04-16T08:20:18.345Z] } [2026-04-16T08:20:18.345Z] ok: [an-a01n02] => { [2026-04-16T08:20:18.345Z] "ansible_selinux": { [2026-04-16T08:20:18.345Z] "config_mode": "enforcing", [2026-04-16T08:20:18.345Z] "mode": "enforcing", [2026-04-16T08:20:18.345Z] "policyvers": 33, [2026-04-16T08:20:18.345Z] "status": "enabled", [2026-04-16T08:20:18.345Z] "type": "targeted" [2026-04-16T08:20:18.345Z] }, [2026-04-16T08:20:18.345Z] "failed_when_result": false [2026-04-16T08:20:18.345Z] } [2026-04-16T08:20:18.345Z] ok: [an-a01dr01] => { [2026-04-16T08:20:18.345Z] "ansible_selinux": { [2026-04-16T08:20:18.345Z] "config_mode": "enforcing", [2026-04-16T08:20:18.345Z] "mode": "enforcing", [2026-04-16T08:20:18.345Z] "policyvers": 33, [2026-04-16T08:20:18.345Z] "status": "enabled", [2026-04-16T08:20:18.345Z] "type": "targeted" [2026-04-16T08:20:18.345Z] }, [2026-04-16T08:20:18.345Z] "failed_when_result": false [2026-04-16T08:20:18.345Z] } [2026-04-16T08:20:18.345Z] [2026-04-16T08:20:18.345Z] PLAY [Create random file on striker01] ***************************************** [2026-04-16T08:20:18.345Z] [2026-04-16T08:20:18.345Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:20:20.173Z] ok: [an-striker01] [2026-04-16T08:20:20.173Z] [2026-04-16T08:20:20.173Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-04-16T08:20:22.000Z] changed: [an-striker01] [2026-04-16T08:20:22.000Z] [2026-04-16T08:20:22.000Z] PLAY [Check archive file propagation] ****************************************** [2026-04-16T08:20:22.000Z] [2026-04-16T08:20:22.000Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:20:24.121Z] ok: [an-a01n01] [2026-04-16T08:20:24.121Z] ok: [an-a01dr01] [2026-04-16T08:20:24.121Z] ok: [an-a01n02] [2026-04-16T08:20:24.121Z] ok: [an-striker02] [2026-04-16T08:20:24.121Z] ok: [an-striker01] [2026-04-16T08:20:24.121Z] [2026-04-16T08:20:24.121Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-04-16T08:20:24.121Z] exit 1 [2026-04-16T08:20:24.121Z] else [2026-04-16T08:20:24.121Z] exit 0 [2026-04-16T08:20:24.121Z] fi [2026-04-16T08:20:24.121Z] ] *** [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-16T08:21:48.246Z] ok: [an-striker01] [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T08:21:48.246Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-16T08:21:48.246Z] ok: [an-striker02] [2026-04-16T08:21:48.246Z] ok: [an-a01n02] [2026-04-16T08:21:48.246Z] ok: [an-a01n01] [2026-04-16T08:21:48.246Z] ok: [an-a01dr01] [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] PLAY [test fencing] ************************************************************ [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:21:48.246Z] ok: [an-a01n02] [2026-04-16T08:21:48.246Z] ok: [an-a01n01] [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T08:21:48.246Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T08:21:48.246Z] ] *** [2026-04-16T08:21:48.246Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T08:21:48.246Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T08:21:48.246Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T08:21:48.246Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-04-16T08:21:48.246Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T08:21:48.246Z] ] ********** [2026-04-16T08:21:48.246Z] skipping: [an-a01n02] [2026-04-16T08:21:48.246Z] changed: [an-a01n01] [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T08:21:48.246Z] skipping: [an-a01n02] [2026-04-16T08:21:48.246Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-04-16T08:21:48.246Z] [2026-04-16T08:21:48.246Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T08:24:58.208Z] ok: [an-a01n01 -> localhost] [2026-04-16T08:24:58.208Z] [2026-04-16T08:24:58.208Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T08:24:58.208Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T08:24:58.208Z] [2026-04-16T08:24:58.208Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T08:24:58.208Z] ] *** [2026-04-16T08:25:00.685Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T08:25:00.685Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T08:25:00.685Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T08:25:00.685Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T08:25:00.685Z] [2026-04-16T08:25:00.685Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-04-16T08:25:00.685Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T08:25:00.685Z] [2026-04-16T08:25:00.685Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T08:25:00.685Z] ] ********** [2026-04-16T08:25:01.477Z] skipping: [an-a01n01] [2026-04-16T08:25:01.477Z] changed: [an-a01n02] [2026-04-16T08:25:01.477Z] [2026-04-16T08:25:01.477Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T08:25:01.477Z] skipping: [an-a01n01] [2026-04-16T08:25:01.477Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-04-16T08:25:01.477Z] [2026-04-16T08:25:01.477Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T08:28:11.917Z] ok: [an-a01n02 -> localhost] [2026-04-16T08:28:11.917Z] [2026-04-16T08:28:11.917Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T08:28:11.917Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T08:28:11.917Z] [2026-04-16T08:28:11.917Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T08:28:11.917Z] ] *** [2026-04-16T08:28:15.992Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T08:28:15.992Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T08:28:15.992Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T08:28:15.992Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T08:28:15.992Z] [2026-04-16T08:28:15.992Z] PLAY [advanced server tests] *************************************************** [2026-04-16T08:28:15.992Z] [2026-04-16T08:28:15.992Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T08:28:17.823Z] ok: [an-a01n01] [2026-04-16T08:28:17.823Z] ok: [an-a01n02] [2026-04-16T08:28:17.823Z] [2026-04-16T08:28:17.823Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T08:28:17.823Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T08:28:17.823Z] [2026-04-16T08:28:17.823Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T08:28:17.823Z] ] *** [2026-04-16T08:28:21.473Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T08:28:21.473Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T08:28:21.473Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T08:28:21.473Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T08:28:21.473Z] [2026-04-16T08:28:21.473Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T08:28:21.473Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T08:28:21.473Z] [2026-04-16T08:28:21.473Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T08:28:21.473Z] if [ -z "$storage" ]; then [2026-04-16T08:28:21.473Z] echo "Unable to detect shared storage" [2026-04-16T08:28:21.473Z] exit 1 [2026-04-16T08:28:21.473Z] fi [2026-04-16T08:28:21.473Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T08:28:21.473Z] ] *** [2026-04-16T08:28:27.760Z] changed: [an-a01n01] [2026-04-16T08:28:27.760Z] changed: [an-a01n02] [2026-04-16T08:28:27.760Z] [2026-04-16T08:28:27.760Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T08:28:27.760Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T08:28:27.760Z] [2026-04-16T08:28:27.760Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T08:28:27.760Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T08:28:27.760Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T08:28:27.760Z] echo "Unable to detect all deploy iso" [2026-04-16T08:28:27.760Z] echo "Detected isos" [2026-04-16T08:28:27.760Z] echo "$isos" [2026-04-16T08:28:27.760Z] exit 1 [2026-04-16T08:28:27.760Z] fi [2026-04-16T08:28:27.760Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T08:28:27.760Z] if [ -z "$installiso" ]; then [2026-04-16T08:28:27.760Z] echo "Unable to detect install iso" [2026-04-16T08:28:27.760Z] echo "Detected isos" [2026-04-16T08:28:27.761Z] echo "$isos" [2026-04-16T08:28:27.761Z] exit 1 [2026-04-16T08:28:27.761Z] fi [2026-04-16T08:28:27.761Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T08:28:27.761Z] ] *** [2026-04-16T08:28:32.887Z] changed: [an-a01n02] [2026-04-16T08:28:32.887Z] changed: [an-a01n01] [2026-04-16T08:28:32.887Z] [2026-04-16T08:28:32.887Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-04-16T08:28:32.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n02, an-a01n01 => (item=1) [2026-04-16T08:28:32.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n02, an-a01n01 => (item=2) [2026-04-16T08:28:32.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n02, an-a01n01 => (item=3) [2026-04-16T08:28:32.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n02, an-a01n01 => (item=4) [2026-04-16T08:28:32.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n02, an-a01n01 => (item=5) [2026-04-16T08:28:32.887Z] [2026-04-16T08:28:32.887Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T08:28:32.887Z] ] *** [2026-04-16T08:28:34.162Z] skipping: [an-a01n02] [2026-04-16T08:28:34.162Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.849672", "end": "2026-04-16 04:28:33.623290", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 04:28:32.773618", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T08:28:34.162Z] ...ignoring [2026-04-16T08:28:34.162Z] [2026-04-16T08:28:34.162Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T08:28:34.162Z] --ci-test \ [2026-04-16T08:28:34.162Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T08:28:34.162Z] --os almalinux9 \ [2026-04-16T08:28:34.162Z] --cpu 2 --ram 4G \ [2026-04-16T08:28:34.162Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T08:28:34.162Z] --install-media {{ installisouuid }} \ [2026-04-16T08:28:34.162Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T08:28:34.162Z] ] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] changed: [an-a01n01] [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n01] [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n01] [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n01] [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n01] [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T08:28:44.256Z] ] *** [2026-04-16T08:28:44.256Z] skipping: [an-a01n02] [2026-04-16T08:28:44.256Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.875102", "end": "2026-04-16 04:28:43.999754", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 04:28:43.124652", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T08:28:44.256Z] ...ignoring [2026-04-16T08:28:44.256Z] [2026-04-16T08:28:44.256Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T08:28:44.256Z] --ci-test \ [2026-04-16T08:28:44.256Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T08:28:44.256Z] --os almalinux9 \ [2026-04-16T08:28:44.256Z] --cpu 2 --ram 4G \ [2026-04-16T08:28:44.256Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T08:28:44.256Z] --install-media {{ installisouuid }} \ [2026-04-16T08:28:44.256Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T08:28:44.256Z] ] *** [2026-04-16T08:28:53.618Z] skipping: [an-a01n02] [2026-04-16T08:28:53.618Z] changed: [an-a01n01] [2026-04-16T08:28:53.618Z] [2026-04-16T08:28:53.618Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T08:28:53.618Z] skipping: [an-a01n01] [2026-04-16T08:28:53.618Z] skipping: [an-a01n02] [2026-04-16T08:28:53.618Z] [2026-04-16T08:28:53.618Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T08:28:53.618Z] skipping: [an-a01n01] [2026-04-16T08:28:53.618Z] skipping: [an-a01n02] [2026-04-16T08:28:53.618Z] [2026-04-16T08:28:53.618Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T08:28:53.618Z] skipping: [an-a01n01] [2026-04-16T08:28:53.618Z] skipping: [an-a01n02] [2026-04-16T08:28:53.618Z] [2026-04-16T08:28:53.618Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T08:28:53.618Z] skipping: [an-a01n01] [2026-04-16T08:28:53.618Z] skipping: [an-a01n02] [2026-04-16T08:28:53.618Z] [2026-04-16T08:28:53.618Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T08:28:53.618Z] ] *** [2026-04-16T08:28:54.892Z] skipping: [an-a01n02] [2026-04-16T08:28:54.892Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.983417", "end": "2026-04-16 04:28:54.534893", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 04:28:53.551476", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T08:28:54.892Z] ...ignoring [2026-04-16T08:28:54.892Z] [2026-04-16T08:28:54.892Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T08:28:54.892Z] --ci-test \ [2026-04-16T08:28:54.892Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T08:28:54.892Z] --os almalinux9 \ [2026-04-16T08:28:54.892Z] --cpu 2 --ram 4G \ [2026-04-16T08:28:54.892Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T08:28:54.892Z] --install-media {{ installisouuid }} \ [2026-04-16T08:28:54.892Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T08:28:54.892Z] ] *** [2026-04-16T08:29:04.256Z] skipping: [an-a01n02] [2026-04-16T08:29:04.256Z] changed: [an-a01n01] [2026-04-16T08:29:04.256Z] [2026-04-16T08:29:04.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T08:29:04.256Z] skipping: [an-a01n01] [2026-04-16T08:29:04.256Z] skipping: [an-a01n02] [2026-04-16T08:29:04.256Z] [2026-04-16T08:29:04.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T08:29:04.256Z] skipping: [an-a01n01] [2026-04-16T08:29:04.256Z] skipping: [an-a01n02] [2026-04-16T08:29:04.256Z] [2026-04-16T08:29:04.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T08:29:04.256Z] skipping: [an-a01n01] [2026-04-16T08:29:04.256Z] skipping: [an-a01n02] [2026-04-16T08:29:04.256Z] [2026-04-16T08:29:04.256Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T08:29:04.256Z] skipping: [an-a01n01] [2026-04-16T08:29:04.256Z] skipping: [an-a01n02] [2026-04-16T08:29:04.256Z] [2026-04-16T08:29:04.256Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T08:29:04.256Z] ] *** [2026-04-16T08:29:05.533Z] skipping: [an-a01n02] [2026-04-16T08:29:05.533Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.917678", "end": "2026-04-16 04:29:04.945893", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 04:29:04.028215", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T08:29:05.533Z] ...ignoring [2026-04-16T08:29:05.533Z] [2026-04-16T08:29:05.533Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T08:29:05.533Z] --ci-test \ [2026-04-16T08:29:05.533Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T08:29:05.533Z] --os almalinux9 \ [2026-04-16T08:29:05.533Z] --cpu 2 --ram 4G \ [2026-04-16T08:29:05.533Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T08:29:05.533Z] --install-media {{ installisouuid }} \ [2026-04-16T08:29:05.533Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T08:29:05.533Z] ] *** [2026-04-16T08:29:14.948Z] skipping: [an-a01n02] [2026-04-16T08:29:14.948Z] changed: [an-a01n01] [2026-04-16T08:29:14.948Z] [2026-04-16T08:29:14.949Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T08:29:14.949Z] skipping: [an-a01n01] [2026-04-16T08:29:14.949Z] skipping: [an-a01n02] [2026-04-16T08:29:14.949Z] [2026-04-16T08:29:14.949Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T08:29:14.949Z] skipping: [an-a01n01] [2026-04-16T08:29:14.949Z] skipping: [an-a01n02] [2026-04-16T08:29:14.949Z] [2026-04-16T08:29:14.949Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T08:29:14.949Z] skipping: [an-a01n01] [2026-04-16T08:29:14.949Z] skipping: [an-a01n02] [2026-04-16T08:29:14.949Z] [2026-04-16T08:29:14.949Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T08:29:14.949Z] skipping: [an-a01n01] [2026-04-16T08:29:14.949Z] skipping: [an-a01n02] [2026-04-16T08:29:14.949Z] [2026-04-16T08:29:14.949Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T08:29:14.949Z] ] *** [2026-04-16T08:29:16.776Z] skipping: [an-a01n02] [2026-04-16T08:29:16.776Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.950808", "end": "2026-04-16 04:29:16.267993", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 04:29:15.317185", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T08:29:16.776Z] ...ignoring [2026-04-16T08:29:16.776Z] [2026-04-16T08:29:16.776Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T08:29:16.776Z] --ci-test \ [2026-04-16T08:29:16.776Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T08:29:16.776Z] --os almalinux9 \ [2026-04-16T08:29:16.776Z] --cpu 2 --ram 4G \ [2026-04-16T08:29:16.776Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T08:29:16.776Z] --install-media {{ installisouuid }} \ [2026-04-16T08:29:16.776Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T08:29:16.776Z] ] *** [2026-04-16T08:29:28.112Z] skipping: [an-a01n02] [2026-04-16T08:29:28.112Z] changed: [an-a01n01] [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T08:29:28.112Z] skipping: [an-a01n01] [2026-04-16T08:29:28.112Z] skipping: [an-a01n02] [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T08:29:28.112Z] skipping: [an-a01n01] [2026-04-16T08:29:28.112Z] skipping: [an-a01n02] [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T08:29:28.112Z] skipping: [an-a01n01] [2026-04-16T08:29:28.112Z] skipping: [an-a01n02] [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T08:29:28.112Z] skipping: [an-a01n01] [2026-04-16T08:29:28.112Z] skipping: [an-a01n02] [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-04-16T08:29:28.112Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T08:29:28.112Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T08:29:28.112Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T08:29:28.112Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T08:29:28.112Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T08:29:28.112Z] [2026-04-16T08:29:28.112Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T08:29:28.112Z] ] *** [2026-04-16T08:29:56.584Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-16T08:29:56.584Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-16T08:29:56.584Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-16T08:29:56.584Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-16T08:29:56.584Z] ok: [an-a01n01] [2026-04-16T08:29:56.584Z] ok: [an-a01n02] [2026-04-16T08:29:56.584Z] [2026-04-16T08:29:56.584Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T08:29:56.584Z] ] *** [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-16T08:32:06.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-16T08:32:06.068Z] ok: [an-a01n02] [2026-04-16T08:32:06.068Z] ok: [an-a01n01] [2026-04-16T08:32:06.068Z] [2026-04-16T08:32:06.068Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T08:32:06.068Z] ] *** [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-16T08:34:46.761Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-16T08:34:46.762Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-16T08:34:46.762Z] ok: [an-a01n02] [2026-04-16T08:34:46.762Z] ok: [an-a01n01] [2026-04-16T08:34:46.762Z] [2026-04-16T08:34:46.762Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T08:34:46.762Z] ] *** [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-04-16T08:37:41.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-04-16T08:37:41.672Z] ok: [an-a01n02] [2026-04-16T08:37:41.672Z] ok: [an-a01n01] [2026-04-16T08:37:41.672Z] [2026-04-16T08:37:41.672Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T08:37:41.672Z] ] *** [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-04-16T08:40:51.683Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-04-16T08:40:51.683Z] ok: [an-a01n02] [2026-04-16T08:40:51.683Z] ok: [an-a01n01] [2026-04-16T08:40:51.683Z] [2026-04-16T08:40:51.683Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-04-16T08:40:51.683Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T08:40:51.683Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T08:40:51.683Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T08:40:51.683Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T08:40:51.683Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T08:40:51.683Z] [2026-04-16T08:40:51.683Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T08:40:51.683Z] ] *** [2026-04-16T08:40:51.683Z] ok: [an-a01n02] [2026-04-16T08:40:51.683Z] ok: [an-a01n01] [2026-04-16T08:40:51.683Z] [2026-04-16T08:40:51.683Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T08:40:51.683Z] ] *** [2026-04-16T08:40:51.683Z] ok: [an-a01n02] [2026-04-16T08:40:51.683Z] ok: [an-a01n01] [2026-04-16T08:40:51.683Z] [2026-04-16T08:40:51.683Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T08:40:51.683Z] ] *** [2026-04-16T08:40:51.683Z] ok: [an-a01n02] [2026-04-16T08:40:51.683Z] ok: [an-a01n01] [2026-04-16T08:40:51.683Z] [2026-04-16T08:40:51.683Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T08:40:51.683Z] ] *** [2026-04-16T08:40:52.960Z] ok: [an-a01n02] [2026-04-16T08:40:52.960Z] ok: [an-a01n01] [2026-04-16T08:40:52.960Z] [2026-04-16T08:40:52.960Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T08:40:52.960Z] ] *** [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-16T08:41:27.094Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-16T08:41:27.094Z] ok: [an-a01n02] [2026-04-16T08:41:27.094Z] ok: [an-a01n01] [2026-04-16T08:41:27.094Z] [2026-04-16T08:41:27.094Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-04-16T08:41:27.094Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T08:41:27.094Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T08:41:27.094Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T08:41:27.094Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T08:41:27.094Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T08:41:27.094Z] [2026-04-16T08:41:27.094Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T08:41:27.094Z] ] *** [2026-04-16T08:41:27.894Z] ok: [an-a01n02] [2026-04-16T08:41:27.894Z] ok: [an-a01n01] [2026-04-16T08:41:27.894Z] [2026-04-16T08:41:27.894Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T08:41:27.894Z] ] *** [2026-04-16T08:41:29.756Z] ok: [an-a01n02] [2026-04-16T08:41:29.756Z] ok: [an-a01n01] [2026-04-16T08:41:29.756Z] [2026-04-16T08:41:29.756Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T08:41:29.756Z] ] *** [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-04-16T08:43:08.977Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-04-16T08:43:08.977Z] ok: [an-a01n02] [2026-04-16T08:43:08.977Z] ok: [an-a01n01] [2026-04-16T08:43:08.977Z] [2026-04-16T08:43:08.977Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T08:43:08.977Z] ] *** [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-16T08:46:03.826Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-16T08:46:03.826Z] ok: [an-a01n02] [2026-04-16T08:46:03.826Z] ok: [an-a01n01] [2026-04-16T08:46:03.826Z] [2026-04-16T08:46:03.826Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T08:46:03.826Z] ] *** [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-04-16T08:49:14.015Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-04-16T08:49:14.015Z] ok: [an-a01n01] [2026-04-16T08:49:14.015Z] ok: [an-a01n02] [2026-04-16T08:49:14.015Z] [2026-04-16T08:49:14.015Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-04-16T08:49:14.015Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T08:49:14.015Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T08:49:14.015Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T08:49:14.015Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T08:49:14.015Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T08:49:14.015Z] [2026-04-16T08:49:14.015Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T08:49:14.015Z] ] *** [2026-04-16T08:49:14.015Z] ok: [an-a01n02] [2026-04-16T08:49:14.015Z] ok: [an-a01n01] [2026-04-16T08:49:14.015Z] [2026-04-16T08:49:14.015Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T08:49:14.015Z] ] *** [2026-04-16T08:49:37.756Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-04-16T08:49:37.756Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-04-16T08:49:37.756Z] ok: [an-a01n02] [2026-04-16T08:49:37.756Z] ok: [an-a01n01] [2026-04-16T08:49:37.756Z] [2026-04-16T08:49:37.756Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T08:49:37.756Z] ] *** [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (344 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (344 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (343 retries left). [2026-04-16T08:53:17.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (343 retries left). [2026-04-16T08:53:17.889Z] ok: [an-a01n02] [2026-04-16T08:53:17.889Z] ok: [an-a01n01] [2026-04-16T08:53:17.889Z] [2026-04-16T08:53:17.889Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T08:53:17.889Z] ] *** [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-16T08:55:27.553Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-16T08:55:27.553Z] ok: [an-a01n02] [2026-04-16T08:55:27.553Z] ok: [an-a01n01] [2026-04-16T08:55:27.553Z] [2026-04-16T08:55:27.553Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T08:55:27.553Z] ] *** [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-04-16T08:57:52.093Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-04-16T08:57:52.093Z] ok: [an-a01n02] [2026-04-16T08:57:52.093Z] ok: [an-a01n01] [2026-04-16T08:57:52.093Z] [2026-04-16T08:57:52.093Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T08:57:52.093Z] [2026-04-16T08:57:52.093Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-04-16T08:57:52.093Z] ok: [an-a01n01] [2026-04-16T08:57:52.093Z] ok: [an-a01n02] [2026-04-16T08:57:52.093Z] [2026-04-16T08:57:52.093Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T08:57:52.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T08:57:52.093Z] [2026-04-16T08:57:52.093Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T08:57:52.093Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T08:57:52.093Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T08:57:52.093Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T08:57:52.093Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T08:57:52.093Z] echo "pcs:" [2026-04-16T08:57:52.093Z] echo "$pcsstatus" [2026-04-16T08:57:52.093Z] exit 1 [2026-04-16T08:57:52.093Z] fi [2026-04-16T08:57:52.093Z] exit 0 [2026-04-16T08:57:52.093Z] ] *** [2026-04-16T08:57:52.093Z] ok: [an-a01n02] [2026-04-16T08:57:52.093Z] ok: [an-a01n01] [2026-04-16T08:57:52.093Z] [2026-04-16T08:57:52.093Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T08:57:52.093Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T08:57:52.093Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T08:57:52.093Z] ret=0 [2026-04-16T08:57:52.093Z] if [ "$servers" != "1" ]; then [2026-04-16T08:57:52.093Z] echo "Not all servers are running on the same node!" [2026-04-16T08:57:52.093Z] echo "$pcsstatus" [2026-04-16T08:57:52.093Z] ret=1 [2026-04-16T08:57:52.093Z] fi [2026-04-16T08:57:52.093Z] echo "$hypervisor" [2026-04-16T08:57:52.093Z] exit $ret [2026-04-16T08:57:52.093Z] ] *** [2026-04-16T08:57:53.369Z] ok: [an-a01n02] [2026-04-16T08:57:53.369Z] ok: [an-a01n01] [2026-04-16T08:57:53.369Z] [2026-04-16T08:57:53.369Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T08:57:53.741Z] ok: [an-a01n01] [2026-04-16T08:57:53.741Z] ok: [an-a01n02] [2026-04-16T08:57:53.741Z] [2026-04-16T08:57:53.741Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T08:57:53.741Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T08:57:53.741Z] [2026-04-16T08:57:53.741Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T08:57:53.741Z] ] ********** [2026-04-16T08:57:54.532Z] skipping: [an-a01n02] [2026-04-16T08:57:54.532Z] changed: [an-a01n01] [2026-04-16T08:57:54.532Z] [2026-04-16T08:57:54.532Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T08:57:54.532Z] skipping: [an-a01n01] [2026-04-16T08:57:54.532Z] skipping: [an-a01n02] [2026-04-16T08:57:54.532Z] [2026-04-16T08:57:54.532Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T08:57:54.532Z] skipping: [an-a01n01] [2026-04-16T08:57:54.532Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T08:57:54.532Z] [2026-04-16T08:57:54.532Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T08:57:54.532Z] ] *** [2026-04-16T08:59:04.876Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T08:59:04.876Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T08:59:04.876Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T08:59:04.876Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T08:59:04.876Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T08:59:04.876Z] ok: [an-a01n02] [2026-04-16T08:59:04.876Z] [2026-04-16T08:59:04.876Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T08:59:04.876Z] skipping: [an-a01n01] [2026-04-16T08:59:04.876Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T08:59:04.876Z] [2026-04-16T08:59:04.876Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T08:59:04.876Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T08:59:04.876Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T08:59:04.876Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T08:59:04.876Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T08:59:04.876Z] echo "pcs:" [2026-04-16T08:59:04.876Z] echo "$pcsstatus" [2026-04-16T08:59:04.876Z] exit 1 [2026-04-16T08:59:04.876Z] fi [2026-04-16T08:59:04.876Z] exit 0 [2026-04-16T08:59:04.876Z] ] *** [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-04-16T09:00:59.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (112 retries left). [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:00:59.150Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:00:59.150Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:00:59.150Z] ret=0 [2026-04-16T09:00:59.150Z] if [ "$servers" != "1" ]; then [2026-04-16T09:00:59.150Z] echo "Not all servers are running on the same node!" [2026-04-16T09:00:59.150Z] echo "$pcsstatus" [2026-04-16T09:00:59.150Z] ret=1 [2026-04-16T09:00:59.150Z] fi [2026-04-16T09:00:59.150Z] echo "$hypervisor" [2026-04-16T09:00:59.150Z] exit $ret [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] => (item=1) [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] => (item=2) [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] => (item=3) [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] => (item=4) [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] => (item=5) [2026-04-16T09:00:59.150Z] skipping: [an-a01n01] [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:00:59.150Z] ] *** [2026-04-16T09:00:59.150Z] ok: [an-a01n02] [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T09:00:59.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:00:59.150Z] [2026-04-16T09:00:59.150Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T09:04:09.053Z] ok: [an-a01n01 -> localhost] [2026-04-16T09:04:09.053Z] ok: [an-a01n02 -> localhost] [2026-04-16T09:04:09.053Z] [2026-04-16T09:04:09.053Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T09:04:09.424Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:04:09.424Z] [2026-04-16T09:04:09.424Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:04:09.424Z] ] *** [2026-04-16T09:04:14.582Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:04:14.582Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:04:14.582Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:04:14.582Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:04:14.582Z] [2026-04-16T09:04:14.582Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:04:14.582Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=1) [2026-04-16T09:04:14.582Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=2) [2026-04-16T09:04:14.582Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=3) [2026-04-16T09:04:14.582Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=4) [2026-04-16T09:04:14.582Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=5) [2026-04-16T09:04:14.582Z] [2026-04-16T09:04:14.582Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:04:14.582Z] ] *** [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-04-16T09:05:25.225Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:05:25.225Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:05:25.225Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:05:25.225Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:05:25.225Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:05:25.225Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:05:25.225Z] echo "pcs:" [2026-04-16T09:05:25.225Z] echo "$pcsstatus" [2026-04-16T09:05:25.225Z] exit 1 [2026-04-16T09:05:25.225Z] fi [2026-04-16T09:05:25.225Z] exit 0 [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:25.225Z] ok: [an-a01n01] [2026-04-16T09:05:25.225Z] ok: [an-a01n02] [2026-04-16T09:05:25.225Z] [2026-04-16T09:05:25.225Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:05:25.225Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:05:25.225Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:05:25.225Z] ret=0 [2026-04-16T09:05:25.225Z] if [ "$servers" != "1" ]; then [2026-04-16T09:05:25.225Z] echo "Not all servers are running on the same node!" [2026-04-16T09:05:25.225Z] echo "$pcsstatus" [2026-04-16T09:05:25.225Z] ret=1 [2026-04-16T09:05:25.225Z] fi [2026-04-16T09:05:25.225Z] echo "$hypervisor" [2026-04-16T09:05:25.225Z] exit $ret [2026-04-16T09:05:25.225Z] ] *** [2026-04-16T09:05:27.051Z] ok: [an-a01n01] [2026-04-16T09:05:27.051Z] ok: [an-a01n02] [2026-04-16T09:05:27.051Z] [2026-04-16T09:05:27.051Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T09:05:27.051Z] ok: [an-a01n01] [2026-04-16T09:05:27.051Z] ok: [an-a01n02] [2026-04-16T09:05:27.051Z] [2026-04-16T09:05:27.051Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T09:05:27.051Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:05:27.051Z] [2026-04-16T09:05:27.051Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T09:05:27.051Z] ] ********** [2026-04-16T09:05:27.845Z] skipping: [an-a01n01] [2026-04-16T09:05:27.845Z] changed: [an-a01n02] [2026-04-16T09:05:27.845Z] [2026-04-16T09:05:27.845Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T09:05:27.845Z] skipping: [an-a01n01] [2026-04-16T09:05:27.845Z] skipping: [an-a01n02] [2026-04-16T09:05:27.845Z] [2026-04-16T09:05:27.845Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T09:05:27.845Z] skipping: [an-a01n02] [2026-04-16T09:05:27.845Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-16T09:05:27.845Z] [2026-04-16T09:05:27.846Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T09:05:27.846Z] ] *** [2026-04-16T09:06:38.593Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-16T09:06:38.593Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-16T09:06:38.593Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-16T09:06:38.593Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-16T09:06:38.593Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-16T09:06:38.593Z] ok: [an-a01n01] [2026-04-16T09:06:38.593Z] [2026-04-16T09:06:38.593Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:06:38.593Z] skipping: [an-a01n02] [2026-04-16T09:06:38.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-16T09:06:38.593Z] [2026-04-16T09:06:38.593Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:06:38.593Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:06:38.593Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:06:38.593Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:06:38.593Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:06:38.593Z] echo "pcs:" [2026-04-16T09:06:38.593Z] echo "$pcsstatus" [2026-04-16T09:06:38.593Z] exit 1 [2026-04-16T09:06:38.593Z] fi [2026-04-16T09:06:38.593Z] exit 0 [2026-04-16T09:06:38.593Z] ] *** [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-16T09:08:02.618Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-04-16T09:08:02.618Z] ok: [an-a01n01] [2026-04-16T09:08:02.618Z] [2026-04-16T09:08:02.618Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:08:02.618Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:08:02.618Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:08:02.618Z] ret=0 [2026-04-16T09:08:02.618Z] if [ "$servers" != "1" ]; then [2026-04-16T09:08:02.618Z] echo "Not all servers are running on the same node!" [2026-04-16T09:08:02.618Z] echo "$pcsstatus" [2026-04-16T09:08:02.618Z] ret=1 [2026-04-16T09:08:02.618Z] fi [2026-04-16T09:08:02.618Z] echo "$hypervisor" [2026-04-16T09:08:02.618Z] exit $ret [2026-04-16T09:08:02.618Z] ] *** [2026-04-16T09:08:02.618Z] ok: [an-a01n01] [2026-04-16T09:08:02.618Z] [2026-04-16T09:08:02.618Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] => (item=1) [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] => (item=2) [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] => (item=3) [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] => (item=4) [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] => (item=5) [2026-04-16T09:08:02.990Z] skipping: [an-a01n02] [2026-04-16T09:08:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-16T09:08:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-16T09:08:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-16T09:08:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-16T09:08:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-16T09:08:02.990Z] [2026-04-16T09:08:02.990Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:08:02.990Z] ] *** [2026-04-16T09:08:19.735Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T09:08:19.735Z] ok: [an-a01n01] [2026-04-16T09:08:19.735Z] [2026-04-16T09:08:19.735Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:08:19.735Z] ] *** [2026-04-16T09:08:19.735Z] ok: [an-a01n01] [2026-04-16T09:08:19.735Z] [2026-04-16T09:08:19.735Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:08:19.735Z] ] *** [2026-04-16T09:08:20.106Z] ok: [an-a01n01] [2026-04-16T09:08:20.106Z] [2026-04-16T09:08:20.106Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:08:20.106Z] ] *** [2026-04-16T09:08:20.906Z] ok: [an-a01n01] [2026-04-16T09:08:20.906Z] [2026-04-16T09:08:20.906Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:08:20.906Z] ] *** [2026-04-16T09:08:21.697Z] ok: [an-a01n01] [2026-04-16T09:08:21.697Z] [2026-04-16T09:08:21.697Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T09:08:22.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:08:22.069Z] [2026-04-16T09:08:22.069Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T09:11:47.090Z] ok: [an-a01n01 -> localhost] [2026-04-16T09:11:47.090Z] ok: [an-a01n02 -> localhost] [2026-04-16T09:11:47.090Z] [2026-04-16T09:11:47.090Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:11:47.090Z] [2026-04-16T09:11:47.090Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:11:47.090Z] ] *** [2026-04-16T09:11:47.090Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:11:47.090Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:11:47.090Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:11:47.090Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:11:47.090Z] [2026-04-16T09:11:47.090Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T09:11:47.090Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T09:11:47.090Z] [2026-04-16T09:11:47.090Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:11:47.090Z] ] *** [2026-04-16T09:12:00.764Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T09:12:00.764Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-04-16T09:12:00.764Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T09:12:00.764Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-04-16T09:12:00.764Z] ok: [an-a01n01] [2026-04-16T09:12:00.764Z] ok: [an-a01n02] [2026-04-16T09:12:00.764Z] [2026-04-16T09:12:00.764Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:12:00.764Z] ] *** [2026-04-16T09:12:00.764Z] ok: [an-a01n02] [2026-04-16T09:12:00.764Z] ok: [an-a01n01] [2026-04-16T09:12:00.764Z] [2026-04-16T09:12:00.764Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:12:00.764Z] ] *** [2026-04-16T09:12:01.135Z] ok: [an-a01n02] [2026-04-16T09:12:01.135Z] ok: [an-a01n01] [2026-04-16T09:12:01.135Z] [2026-04-16T09:12:01.135Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:12:01.135Z] ] *** [2026-04-16T09:12:01.927Z] ok: [an-a01n02] [2026-04-16T09:12:01.927Z] ok: [an-a01n01] [2026-04-16T09:12:01.927Z] [2026-04-16T09:12:01.927Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:12:01.927Z] ] *** [2026-04-16T09:12:02.724Z] ok: [an-a01n02] [2026-04-16T09:12:02.724Z] ok: [an-a01n01] [2026-04-16T09:12:02.724Z] [2026-04-16T09:12:02.724Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-04-16T09:12:02.724Z] ok: [an-a01n01] [2026-04-16T09:12:02.724Z] ok: [an-a01n02] [2026-04-16T09:12:02.724Z] [2026-04-16T09:12:02.724Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:12:02.724Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:12:02.724Z] [2026-04-16T09:12:02.724Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:12:02.724Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:12:02.724Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:12:02.724Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:12:02.724Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:12:02.724Z] echo "pcs:" [2026-04-16T09:12:02.724Z] echo "$pcsstatus" [2026-04-16T09:12:02.724Z] exit 1 [2026-04-16T09:12:02.724Z] fi [2026-04-16T09:12:02.724Z] exit 0 [2026-04-16T09:12:02.724Z] ] *** [2026-04-16T09:12:04.762Z] ok: [an-a01n02] [2026-04-16T09:12:04.762Z] ok: [an-a01n01] [2026-04-16T09:12:04.762Z] [2026-04-16T09:12:04.762Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:12:04.762Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:12:04.762Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:12:04.762Z] ret=0 [2026-04-16T09:12:04.762Z] if [ "$servers" != "1" ]; then [2026-04-16T09:12:04.762Z] echo "Not all servers are running on the same node!" [2026-04-16T09:12:04.762Z] echo "$pcsstatus" [2026-04-16T09:12:04.762Z] ret=1 [2026-04-16T09:12:04.762Z] fi [2026-04-16T09:12:04.762Z] echo "$hypervisor" [2026-04-16T09:12:04.762Z] exit $ret [2026-04-16T09:12:04.762Z] ] *** [2026-04-16T09:12:07.229Z] ok: [an-a01n02] [2026-04-16T09:12:07.229Z] ok: [an-a01n01] [2026-04-16T09:12:07.229Z] [2026-04-16T09:12:07.229Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T09:12:07.229Z] ok: [an-a01n01] [2026-04-16T09:12:07.229Z] ok: [an-a01n02] [2026-04-16T09:12:07.229Z] [2026-04-16T09:12:07.229Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T09:12:07.229Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:12:07.229Z] [2026-04-16T09:12:07.229Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T09:12:07.229Z] ] ********** [2026-04-16T09:12:07.600Z] skipping: [an-a01n02] [2026-04-16T09:12:07.600Z] changed: [an-a01n01] [2026-04-16T09:12:07.600Z] [2026-04-16T09:12:07.600Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T09:12:07.970Z] skipping: [an-a01n01] [2026-04-16T09:12:07.970Z] skipping: [an-a01n02] [2026-04-16T09:12:07.970Z] [2026-04-16T09:12:07.970Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T09:12:07.970Z] skipping: [an-a01n01] [2026-04-16T09:12:07.970Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T09:12:07.970Z] [2026-04-16T09:12:07.970Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T09:12:07.970Z] ] *** [2026-04-16T09:13:18.133Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T09:13:18.133Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T09:13:18.133Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T09:13:18.133Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T09:13:18.133Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T09:13:18.133Z] ok: [an-a01n02] [2026-04-16T09:13:18.133Z] [2026-04-16T09:13:18.133Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:13:18.133Z] skipping: [an-a01n01] [2026-04-16T09:13:18.133Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T09:13:18.133Z] [2026-04-16T09:13:18.133Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:13:18.133Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:13:18.133Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:13:18.133Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:13:18.133Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:13:18.133Z] echo "pcs:" [2026-04-16T09:13:18.133Z] echo "$pcsstatus" [2026-04-16T09:13:18.133Z] exit 1 [2026-04-16T09:13:18.133Z] fi [2026-04-16T09:13:18.133Z] exit 0 [2026-04-16T09:13:18.133Z] ] *** [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T09:14:28.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-16T09:14:28.362Z] ok: [an-a01n02] [2026-04-16T09:14:28.362Z] [2026-04-16T09:14:28.362Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:14:28.362Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:14:28.362Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:14:28.362Z] ret=0 [2026-04-16T09:14:28.362Z] if [ "$servers" != "1" ]; then [2026-04-16T09:14:28.362Z] echo "Not all servers are running on the same node!" [2026-04-16T09:14:28.362Z] echo "$pcsstatus" [2026-04-16T09:14:28.362Z] ret=1 [2026-04-16T09:14:28.362Z] fi [2026-04-16T09:14:28.362Z] echo "$hypervisor" [2026-04-16T09:14:28.362Z] exit $ret [2026-04-16T09:14:28.362Z] ] *** [2026-04-16T09:14:28.362Z] ok: [an-a01n02] [2026-04-16T09:14:28.362Z] [2026-04-16T09:14:28.362Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] => (item=1) [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] => (item=2) [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] => (item=3) [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] => (item=4) [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] => (item=5) [2026-04-16T09:14:28.362Z] skipping: [an-a01n01] [2026-04-16T09:14:28.362Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T09:14:28.362Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T09:14:28.362Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T09:14:28.362Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T09:14:28.362Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T09:14:28.362Z] [2026-04-16T09:14:28.362Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:14:28.362Z] ] *** [2026-04-16T09:14:29.155Z] ok: [an-a01n02] [2026-04-16T09:14:29.155Z] [2026-04-16T09:14:29.155Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:14:29.155Z] ] *** [2026-04-16T09:14:30.429Z] ok: [an-a01n02] [2026-04-16T09:14:30.429Z] [2026-04-16T09:14:30.429Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:14:30.429Z] ] *** [2026-04-16T09:14:31.221Z] ok: [an-a01n02] [2026-04-16T09:14:31.221Z] [2026-04-16T09:14:31.221Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:14:31.221Z] ] *** [2026-04-16T09:14:32.013Z] ok: [an-a01n02] [2026-04-16T09:14:32.013Z] [2026-04-16T09:14:32.013Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:14:32.013Z] ] *** [2026-04-16T09:14:32.805Z] ok: [an-a01n02] [2026-04-16T09:14:32.805Z] [2026-04-16T09:14:32.805Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T09:14:32.805Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:14:32.805Z] [2026-04-16T09:14:32.805Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n02 -> localhost] [2026-04-16T09:17:57.836Z] ok: [an-a01n01 -> localhost] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:17:57.836Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:17:57.836Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:17:57.836Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:17:57.836Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:17:57.836Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:17:57.836Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:17:57.836Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:17:57.836Z] echo "pcs:" [2026-04-16T09:17:57.836Z] echo "$pcsstatus" [2026-04-16T09:17:57.836Z] exit 1 [2026-04-16T09:17:57.836Z] fi [2026-04-16T09:17:57.836Z] exit 0 [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:17:57.836Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:17:57.836Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:17:57.836Z] ret=0 [2026-04-16T09:17:57.836Z] if [ "$servers" != "1" ]; then [2026-04-16T09:17:57.836Z] echo "Not all servers are running on the same node!" [2026-04-16T09:17:57.836Z] echo "$pcsstatus" [2026-04-16T09:17:57.836Z] ret=1 [2026-04-16T09:17:57.836Z] fi [2026-04-16T09:17:57.836Z] echo "$hypervisor" [2026-04-16T09:17:57.836Z] exit $ret [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T09:17:57.836Z] ok: [an-a01n01] [2026-04-16T09:17:57.836Z] ok: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T09:17:57.836Z] ] ********** [2026-04-16T09:17:57.836Z] skipping: [an-a01n01] [2026-04-16T09:17:57.836Z] changed: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T09:17:57.836Z] skipping: [an-a01n01] [2026-04-16T09:17:57.836Z] skipping: [an-a01n02] [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T09:17:57.836Z] skipping: [an-a01n02] [2026-04-16T09:17:57.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-16T09:17:57.836Z] [2026-04-16T09:17:57.836Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T09:17:57.836Z] ] *** [2026-04-16T09:19:08.223Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-16T09:19:08.223Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-16T09:19:08.223Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-16T09:19:08.223Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-16T09:19:08.223Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-16T09:19:08.223Z] ok: [an-a01n01] [2026-04-16T09:19:08.223Z] [2026-04-16T09:19:08.223Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:19:08.223Z] skipping: [an-a01n02] [2026-04-16T09:19:08.223Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-16T09:19:08.223Z] [2026-04-16T09:19:08.223Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:19:08.223Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:19:08.223Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:19:08.223Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:19:08.223Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:19:08.223Z] echo "pcs:" [2026-04-16T09:19:08.223Z] echo "$pcsstatus" [2026-04-16T09:19:08.223Z] exit 1 [2026-04-16T09:19:08.223Z] fi [2026-04-16T09:19:08.223Z] exit 0 [2026-04-16T09:19:08.223Z] ] *** [2026-04-16T09:20:06.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T09:20:06.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T09:20:06.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T09:20:06.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T09:20:06.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T09:20:06.842Z] ok: [an-a01n01] [2026-04-16T09:20:06.842Z] [2026-04-16T09:20:06.842Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:20:06.842Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:20:06.842Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:20:06.842Z] ret=0 [2026-04-16T09:20:06.842Z] if [ "$servers" != "1" ]; then [2026-04-16T09:20:06.842Z] echo "Not all servers are running on the same node!" [2026-04-16T09:20:06.842Z] echo "$pcsstatus" [2026-04-16T09:20:06.842Z] ret=1 [2026-04-16T09:20:06.842Z] fi [2026-04-16T09:20:06.842Z] echo "$hypervisor" [2026-04-16T09:20:06.842Z] exit $ret [2026-04-16T09:20:06.842Z] ] *** [2026-04-16T09:20:06.842Z] ok: [an-a01n01] [2026-04-16T09:20:06.842Z] [2026-04-16T09:20:06.842Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] => (item=1) [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] => (item=2) [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] => (item=3) [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] => (item=4) [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] => (item=5) [2026-04-16T09:20:06.842Z] skipping: [an-a01n02] [2026-04-16T09:20:06.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-16T09:20:06.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-16T09:20:06.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-16T09:20:06.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-16T09:20:06.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-16T09:20:06.842Z] [2026-04-16T09:20:06.842Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:20:06.842Z] ] *** [2026-04-16T09:20:20.542Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T09:20:20.542Z] ok: [an-a01n01] [2026-04-16T09:20:20.542Z] [2026-04-16T09:20:20.542Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:20:20.542Z] ] *** [2026-04-16T09:20:20.542Z] ok: [an-a01n01] [2026-04-16T09:20:20.542Z] [2026-04-16T09:20:20.542Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:20:20.542Z] ] *** [2026-04-16T09:20:20.542Z] ok: [an-a01n01] [2026-04-16T09:20:20.542Z] [2026-04-16T09:20:20.542Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:20:20.542Z] ] *** [2026-04-16T09:20:21.334Z] ok: [an-a01n01] [2026-04-16T09:20:21.334Z] [2026-04-16T09:20:21.334Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:20:21.334Z] ] *** [2026-04-16T09:20:22.125Z] ok: [an-a01n01] [2026-04-16T09:20:22.125Z] [2026-04-16T09:20:22.125Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T09:20:22.125Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:20:22.125Z] [2026-04-16T09:20:22.125Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n01 -> localhost] [2026-04-16T09:23:47.110Z] ok: [an-a01n02 -> localhost] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:23:47.110Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:23:47.110Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:23:47.110Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:23:47.110Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:23:47.110Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:23:47.110Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:23:47.110Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:23:47.110Z] echo "pcs:" [2026-04-16T09:23:47.110Z] echo "$pcsstatus" [2026-04-16T09:23:47.110Z] exit 1 [2026-04-16T09:23:47.110Z] fi [2026-04-16T09:23:47.110Z] exit 0 [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:23:47.110Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:23:47.110Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:23:47.110Z] ret=0 [2026-04-16T09:23:47.110Z] if [ "$servers" != "1" ]; then [2026-04-16T09:23:47.110Z] echo "Not all servers are running on the same node!" [2026-04-16T09:23:47.110Z] echo "$pcsstatus" [2026-04-16T09:23:47.110Z] ret=1 [2026-04-16T09:23:47.110Z] fi [2026-04-16T09:23:47.110Z] echo "$hypervisor" [2026-04-16T09:23:47.110Z] exit $ret [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-04-16T09:23:47.110Z] ok: [an-a01n01] [2026-04-16T09:23:47.110Z] ok: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-04-16T09:23:47.110Z] ] ********** [2026-04-16T09:23:47.110Z] skipping: [an-a01n02] [2026-04-16T09:23:47.110Z] changed: [an-a01n01] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-04-16T09:23:47.110Z] skipping: [an-a01n01] [2026-04-16T09:23:47.110Z] skipping: [an-a01n02] [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-04-16T09:23:47.110Z] skipping: [an-a01n01] [2026-04-16T09:23:47.110Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-16T09:23:47.110Z] [2026-04-16T09:23:47.110Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-04-16T09:23:47.110Z] ] *** [2026-04-16T09:24:57.296Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-16T09:24:57.296Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-16T09:24:57.296Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-16T09:24:57.296Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-16T09:24:57.296Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-16T09:24:57.296Z] ok: [an-a01n02] [2026-04-16T09:24:57.296Z] [2026-04-16T09:24:57.296Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-04-16T09:24:57.296Z] skipping: [an-a01n01] [2026-04-16T09:24:57.296Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-16T09:24:57.296Z] [2026-04-16T09:24:57.296Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:24:57.296Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:24:57.296Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:24:57.296Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:24:57.297Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:24:57.297Z] echo "pcs:" [2026-04-16T09:24:57.297Z] echo "$pcsstatus" [2026-04-16T09:24:57.297Z] exit 1 [2026-04-16T09:24:57.297Z] fi [2026-04-16T09:24:57.297Z] exit 0 [2026-04-16T09:24:57.297Z] ] *** [2026-04-16T09:25:56.039Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-16T09:25:56.039Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-16T09:25:56.039Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-16T09:25:56.039Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-16T09:25:56.039Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-16T09:25:56.039Z] ok: [an-a01n02] [2026-04-16T09:25:56.039Z] [2026-04-16T09:25:56.039Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:25:56.039Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:25:56.039Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:25:56.039Z] ret=0 [2026-04-16T09:25:56.039Z] if [ "$servers" != "1" ]; then [2026-04-16T09:25:56.039Z] echo "Not all servers are running on the same node!" [2026-04-16T09:25:56.039Z] echo "$pcsstatus" [2026-04-16T09:25:56.039Z] ret=1 [2026-04-16T09:25:56.039Z] fi [2026-04-16T09:25:56.039Z] echo "$hypervisor" [2026-04-16T09:25:56.039Z] exit $ret [2026-04-16T09:25:56.039Z] ] *** [2026-04-16T09:25:56.039Z] ok: [an-a01n02] [2026-04-16T09:25:56.039Z] [2026-04-16T09:25:56.039Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] => (item=1) [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] => (item=2) [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] => (item=3) [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] => (item=4) [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] => (item=5) [2026-04-16T09:25:56.039Z] skipping: [an-a01n01] [2026-04-16T09:25:56.039Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-16T09:25:56.039Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-16T09:25:56.039Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-16T09:25:56.039Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-16T09:25:56.039Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-16T09:25:56.039Z] [2026-04-16T09:25:56.039Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:25:56.039Z] ] *** [2026-04-16T09:26:09.711Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-16T09:26:09.711Z] ok: [an-a01n02] [2026-04-16T09:26:09.711Z] [2026-04-16T09:26:09.711Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:26:09.711Z] ] *** [2026-04-16T09:26:09.711Z] ok: [an-a01n02] [2026-04-16T09:26:09.711Z] [2026-04-16T09:26:09.711Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:26:09.711Z] ] *** [2026-04-16T09:26:10.082Z] ok: [an-a01n02] [2026-04-16T09:26:10.082Z] [2026-04-16T09:26:10.082Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:26:10.082Z] ] *** [2026-04-16T09:26:10.875Z] ok: [an-a01n02] [2026-04-16T09:26:10.875Z] [2026-04-16T09:26:10.875Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T09:26:10.875Z] ] *** [2026-04-16T09:26:11.866Z] ok: [an-a01n02] [2026-04-16T09:26:11.866Z] [2026-04-16T09:26:11.866Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-04-16T09:26:11.866Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-16T09:26:11.866Z] [2026-04-16T09:26:11.866Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n01 -> localhost] [2026-04-16T09:29:36.903Z] ok: [an-a01n02 -> localhost] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:29:36.903Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:29:36.903Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:29:36.903Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] PLAY [basic server tests] ****************************************************** [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:29:36.903Z] ok: [an-a01n01] [2026-04-16T09:29:36.903Z] ok: [an-a01n02] [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:36.903Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:29:36.903Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:29:36.903Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:29:36.903Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T09:29:36.903Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:36.903Z] [2026-04-16T09:29:36.903Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T09:29:36.903Z] if [ -z "$storage" ]; then [2026-04-16T09:29:36.903Z] echo "Unable to detect shared storage" [2026-04-16T09:29:36.903Z] exit 1 [2026-04-16T09:29:36.903Z] fi [2026-04-16T09:29:36.903Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:29:36.903Z] ] *** [2026-04-16T09:29:43.433Z] changed: [an-a01n01] [2026-04-16T09:29:43.433Z] changed: [an-a01n02] [2026-04-16T09:29:43.433Z] [2026-04-16T09:29:43.433Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T09:29:43.433Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:43.433Z] [2026-04-16T09:29:43.433Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T09:29:43.433Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T09:29:43.433Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T09:29:43.433Z] echo "Unable to detect all deploy iso" [2026-04-16T09:29:43.433Z] echo "Detected isos" [2026-04-16T09:29:43.433Z] echo "$isos" [2026-04-16T09:29:43.433Z] exit 1 [2026-04-16T09:29:43.433Z] fi [2026-04-16T09:29:43.433Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T09:29:43.433Z] if [ -z "$installiso" ]; then [2026-04-16T09:29:43.433Z] echo "Unable to detect install iso" [2026-04-16T09:29:43.433Z] echo "Detected isos" [2026-04-16T09:29:43.433Z] echo "$isos" [2026-04-16T09:29:43.433Z] exit 1 [2026-04-16T09:29:43.433Z] fi [2026-04-16T09:29:43.433Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:29:43.433Z] ] *** [2026-04-16T09:29:49.721Z] changed: [an-a01n01] [2026-04-16T09:29:49.721Z] changed: [an-a01n02] [2026-04-16T09:29:49.721Z] [2026-04-16T09:29:49.721Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T09:29:49.721Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:49.721Z] [2026-04-16T09:29:49.721Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:29:49.721Z] ] *** [2026-04-16T09:29:50.514Z] skipping: [an-a01n02] [2026-04-16T09:29:50.514Z] ok: [an-a01n01] [2026-04-16T09:29:50.514Z] [2026-04-16T09:29:50.514Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T09:29:50.514Z] --ci-test \ [2026-04-16T09:29:50.514Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T09:29:50.514Z] --os almalinux9 \ [2026-04-16T09:29:50.514Z] --cpu 2 --ram 4G \ [2026-04-16T09:29:50.514Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T09:29:50.514Z] --install-media {{ installisouuid }} \ [2026-04-16T09:29:50.514Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T09:29:50.514Z] ] *** [2026-04-16T09:29:50.514Z] skipping: [an-a01n01] [2026-04-16T09:29:50.514Z] skipping: [an-a01n02] [2026-04-16T09:29:50.514Z] [2026-04-16T09:29:50.514Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T09:29:50.886Z] skipping: [an-a01n01] [2026-04-16T09:29:50.886Z] skipping: [an-a01n02] [2026-04-16T09:29:50.886Z] [2026-04-16T09:29:50.886Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:29:50.886Z] skipping: [an-a01n01] [2026-04-16T09:29:50.886Z] skipping: [an-a01n02] [2026-04-16T09:29:50.886Z] [2026-04-16T09:29:50.886Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T09:29:50.886Z] skipping: [an-a01n01] [2026-04-16T09:29:50.886Z] skipping: [an-a01n02] [2026-04-16T09:29:50.886Z] [2026-04-16T09:29:50.886Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T09:29:50.886Z] skipping: [an-a01n01] [2026-04-16T09:29:50.886Z] skipping: [an-a01n02] [2026-04-16T09:29:50.886Z] [2026-04-16T09:29:50.886Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T09:29:51.258Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:51.258Z] [2026-04-16T09:29:51.258Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:29:51.258Z] ] *** [2026-04-16T09:29:52.531Z] skipping: [an-a01n01] [2026-04-16T09:29:52.531Z] ok: [an-a01n02] [2026-04-16T09:29:52.531Z] [2026-04-16T09:29:52.531Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T09:29:52.531Z] --ci-test \ [2026-04-16T09:29:52.531Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T09:29:52.531Z] --os almalinux9 \ [2026-04-16T09:29:52.531Z] --cpu 2 --ram 4G \ [2026-04-16T09:29:52.531Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T09:29:52.531Z] --install-media {{ installisouuid }} \ [2026-04-16T09:29:52.531Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T09:29:52.531Z] ] *** [2026-04-16T09:29:52.903Z] skipping: [an-a01n01] [2026-04-16T09:29:52.903Z] skipping: [an-a01n02] [2026-04-16T09:29:52.903Z] [2026-04-16T09:29:52.903Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T09:29:52.903Z] skipping: [an-a01n01] [2026-04-16T09:29:52.903Z] skipping: [an-a01n02] [2026-04-16T09:29:52.903Z] [2026-04-16T09:29:52.903Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:29:52.903Z] skipping: [an-a01n01] [2026-04-16T09:29:52.903Z] skipping: [an-a01n02] [2026-04-16T09:29:52.903Z] [2026-04-16T09:29:52.903Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T09:29:52.903Z] skipping: [an-a01n01] [2026-04-16T09:29:52.903Z] skipping: [an-a01n02] [2026-04-16T09:29:52.903Z] [2026-04-16T09:29:52.903Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T09:29:52.903Z] skipping: [an-a01n01] [2026-04-16T09:29:52.903Z] skipping: [an-a01n02] [2026-04-16T09:29:52.903Z] [2026-04-16T09:29:52.903Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T09:29:53.275Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:53.275Z] [2026-04-16T09:29:53.275Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:29:53.275Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:29:53.275Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:29:53.275Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:29:53.275Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:29:53.275Z] echo "pcs:" [2026-04-16T09:29:53.275Z] echo "$pcsstatus" [2026-04-16T09:29:53.275Z] exit 1 [2026-04-16T09:29:53.275Z] fi [2026-04-16T09:29:53.275Z] exit 0 [2026-04-16T09:29:53.275Z] ] *** [2026-04-16T09:29:55.104Z] ok: [an-a01n02] [2026-04-16T09:29:55.104Z] ok: [an-a01n01] [2026-04-16T09:29:55.104Z] [2026-04-16T09:29:55.104Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:29:55.104Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:29:55.104Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:29:55.104Z] ret=0 [2026-04-16T09:29:55.104Z] if [ "$servers" != "1" ]; then [2026-04-16T09:29:55.104Z] echo "Not all servers are running on the same node!" [2026-04-16T09:29:55.104Z] echo "$pcsstatus" [2026-04-16T09:29:55.104Z] ret=1 [2026-04-16T09:29:55.104Z] fi [2026-04-16T09:29:55.104Z] echo "$hypervisor" [2026-04-16T09:29:55.104Z] exit $ret [2026-04-16T09:29:55.104Z] ] *** [2026-04-16T09:29:57.571Z] ok: [an-a01n01] [2026-04-16T09:29:57.571Z] ok: [an-a01n02] [2026-04-16T09:29:57.571Z] [2026-04-16T09:29:57.571Z] PLAY [delete extra server tests] *********************************************** [2026-04-16T09:29:57.571Z] [2026-04-16T09:29:57.571Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:29:59.629Z] ok: [an-a01n01] [2026-04-16T09:29:59.629Z] ok: [an-a01n02] [2026-04-16T09:29:59.629Z] [2026-04-16T09:29:59.629Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T09:29:59.629Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:29:59.629Z] [2026-04-16T09:29:59.629Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:29:59.629Z] ] *** [2026-04-16T09:30:03.701Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:30:03.701Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:30:03.701Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:30:03.701Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:30:03.701Z] [2026-04-16T09:30:03.702Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-04-16T09:30:03.702Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T09:30:03.702Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T09:30:03.702Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T09:30:03.702Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T09:30:03.702Z] [2026-04-16T09:30:03.702Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:30:03.702Z] ] *** [2026-04-16T09:30:05.529Z] skipping: [an-a01n01] [2026-04-16T09:30:05.529Z] ok: [an-a01n02] [2026-04-16T09:30:05.529Z] [2026-04-16T09:30:05.529Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T09:30:05.529Z] --force \ [2026-04-16T09:30:05.529Z] --server an-test-deploy{{ servernum }} [2026-04-16T09:30:05.529Z] ] *** [2026-04-16T09:30:14.883Z] skipping: [an-a01n01] [2026-04-16T09:30:14.883Z] changed: [an-a01n02] [2026-04-16T09:30:14.883Z] [2026-04-16T09:30:14.883Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:30:14.883Z] ] *** [2026-04-16T09:30:38.842Z] skipping: [an-a01n01] [2026-04-16T09:30:38.842Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-04-16T09:30:38.842Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-04-16T09:30:38.842Z] ok: [an-a01n02] [2026-04-16T09:30:38.842Z] [2026-04-16T09:30:38.842Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:30:38.842Z] ] *** [2026-04-16T09:30:38.842Z] skipping: [an-a01n01] [2026-04-16T09:30:38.842Z] ok: [an-a01n02] [2026-04-16T09:30:38.842Z] [2026-04-16T09:30:38.842Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:30:38.842Z] ] *** [2026-04-16T09:30:39.633Z] skipping: [an-a01n01] [2026-04-16T09:30:39.633Z] ok: [an-a01n02] [2026-04-16T09:30:39.633Z] [2026-04-16T09:30:39.633Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:30:39.633Z] ] *** [2026-04-16T09:30:41.460Z] skipping: [an-a01n01] [2026-04-16T09:30:41.460Z] ok: [an-a01n02] [2026-04-16T09:30:41.460Z] [2026-04-16T09:30:41.460Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T09:30:41.460Z] --force \ [2026-04-16T09:30:41.460Z] --server an-test-deploy{{ servernum }} [2026-04-16T09:30:41.460Z] ] *** [2026-04-16T09:30:49.148Z] skipping: [an-a01n01] [2026-04-16T09:30:49.148Z] changed: [an-a01n02] [2026-04-16T09:30:49.148Z] [2026-04-16T09:30:49.148Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:30:49.148Z] ] *** [2026-04-16T09:32:13.167Z] skipping: [an-a01n01] [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-04-16T09:32:13.167Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-04-16T09:32:13.167Z] ok: [an-a01n02] [2026-04-16T09:32:13.167Z] [2026-04-16T09:32:13.167Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:32:13.167Z] ] *** [2026-04-16T09:32:13.167Z] skipping: [an-a01n01] [2026-04-16T09:32:13.167Z] ok: [an-a01n02] [2026-04-16T09:32:13.167Z] [2026-04-16T09:32:13.167Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:32:13.167Z] ] *** [2026-04-16T09:32:13.167Z] skipping: [an-a01n01] [2026-04-16T09:32:13.167Z] ok: [an-a01n02] [2026-04-16T09:32:13.167Z] [2026-04-16T09:32:13.167Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:32:13.167Z] ] *** [2026-04-16T09:32:13.167Z] skipping: [an-a01n01] [2026-04-16T09:32:13.167Z] ok: [an-a01n02] [2026-04-16T09:32:13.167Z] [2026-04-16T09:32:13.167Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T09:32:13.167Z] --force \ [2026-04-16T09:32:13.167Z] --server an-test-deploy{{ servernum }} [2026-04-16T09:32:13.167Z] ] *** [2026-04-16T09:32:13.167Z] skipping: [an-a01n01] [2026-04-16T09:32:13.167Z] changed: [an-a01n02] [2026-04-16T09:32:13.167Z] [2026-04-16T09:32:13.167Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:32:13.167Z] ] *** [2026-04-16T09:33:11.753Z] skipping: [an-a01n01] [2026-04-16T09:33:11.753Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-04-16T09:33:11.753Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-04-16T09:33:11.753Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-04-16T09:33:11.753Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-04-16T09:33:11.753Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-04-16T09:33:11.753Z] ok: [an-a01n02] [2026-04-16T09:33:11.753Z] [2026-04-16T09:33:11.753Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:33:11.753Z] ] *** [2026-04-16T09:33:11.753Z] skipping: [an-a01n01] [2026-04-16T09:33:11.753Z] ok: [an-a01n02] [2026-04-16T09:33:11.753Z] [2026-04-16T09:33:11.753Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:33:11.753Z] ] *** [2026-04-16T09:33:11.753Z] skipping: [an-a01n01] [2026-04-16T09:33:11.753Z] ok: [an-a01n02] [2026-04-16T09:33:11.753Z] [2026-04-16T09:33:11.753Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:33:11.753Z] ] *** [2026-04-16T09:33:13.578Z] skipping: [an-a01n01] [2026-04-16T09:33:13.578Z] ok: [an-a01n02] [2026-04-16T09:33:13.578Z] [2026-04-16T09:33:13.578Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T09:33:13.578Z] --force \ [2026-04-16T09:33:13.578Z] --server an-test-deploy{{ servernum }} [2026-04-16T09:33:13.578Z] ] *** [2026-04-16T09:33:23.080Z] skipping: [an-a01n01] [2026-04-16T09:33:23.080Z] changed: [an-a01n02] [2026-04-16T09:33:23.080Z] [2026-04-16T09:33:23.080Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:33:23.080Z] ] *** [2026-04-16T09:36:18.053Z] skipping: [an-a01n01] [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (55 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (54 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (53 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (52 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (51 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (50 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (49 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (48 retries left). [2026-04-16T09:36:18.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (47 retries left). [2026-04-16T09:36:18.053Z] ok: [an-a01n02] [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:36:18.053Z] ] *** [2026-04-16T09:36:18.053Z] skipping: [an-a01n01] [2026-04-16T09:36:18.053Z] ok: [an-a01n02] [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:36:18.053Z] ] *** [2026-04-16T09:36:18.053Z] skipping: [an-a01n01] [2026-04-16T09:36:18.053Z] ok: [an-a01n02] [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] PLAY [create server for DR testing] ******************************************** [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:36:18.053Z] ok: [an-a01n02] [2026-04-16T09:36:18.053Z] ok: [an-a01n01] [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T09:36:18.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:36:18.053Z] ] *** [2026-04-16T09:36:18.053Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:36:18.053Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:36:18.053Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:36:18.053Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T09:36:18.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T09:36:18.053Z] if [ -z "$storage" ]; then [2026-04-16T09:36:18.053Z] echo "Unable to detect shared storage" [2026-04-16T09:36:18.053Z] exit 1 [2026-04-16T09:36:18.053Z] fi [2026-04-16T09:36:18.053Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:36:18.053Z] ] *** [2026-04-16T09:36:18.053Z] changed: [an-a01n02] [2026-04-16T09:36:18.053Z] changed: [an-a01n01] [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T09:36:18.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T09:36:18.053Z] [2026-04-16T09:36:18.053Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T09:36:18.053Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T09:36:18.053Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T09:36:18.053Z] echo "Unable to detect all deploy iso" [2026-04-16T09:36:18.053Z] echo "Detected isos" [2026-04-16T09:36:18.053Z] echo "$isos" [2026-04-16T09:36:18.053Z] exit 1 [2026-04-16T09:36:18.053Z] fi [2026-04-16T09:36:18.053Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T09:36:18.053Z] if [ -z "$installiso" ]; then [2026-04-16T09:36:18.053Z] echo "Unable to detect install iso" [2026-04-16T09:36:18.053Z] echo "Detected isos" [2026-04-16T09:36:18.053Z] echo "$isos" [2026-04-16T09:36:18.053Z] exit 1 [2026-04-16T09:36:18.053Z] fi [2026-04-16T09:36:18.053Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:36:18.053Z] ] *** [2026-04-16T09:36:24.336Z] changed: [an-a01n01] [2026-04-16T09:36:24.336Z] changed: [an-a01n02] [2026-04-16T09:36:24.336Z] [2026-04-16T09:36:24.336Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T09:36:24.336Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T09:36:24.336Z] [2026-04-16T09:36:24.336Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:36:24.336Z] ] *** [2026-04-16T09:36:25.611Z] skipping: [an-a01n02] [2026-04-16T09:36:25.611Z] ok: [an-a01n01] [2026-04-16T09:36:25.611Z] [2026-04-16T09:36:25.611Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T09:36:25.611Z] --ci-test \ [2026-04-16T09:36:25.611Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T09:36:25.611Z] --os almalinux9 \ [2026-04-16T09:36:25.611Z] --cpu 2 --ram 4G \ [2026-04-16T09:36:25.611Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T09:36:25.611Z] --install-media {{ installisouuid }} \ [2026-04-16T09:36:25.611Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T09:36:25.611Z] ] *** [2026-04-16T09:36:25.611Z] skipping: [an-a01n01] [2026-04-16T09:36:25.611Z] skipping: [an-a01n02] [2026-04-16T09:36:25.611Z] [2026-04-16T09:36:25.611Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T09:36:25.611Z] skipping: [an-a01n01] [2026-04-16T09:36:25.611Z] skipping: [an-a01n02] [2026-04-16T09:36:25.611Z] [2026-04-16T09:36:25.611Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:36:25.982Z] skipping: [an-a01n01] [2026-04-16T09:36:25.982Z] skipping: [an-a01n02] [2026-04-16T09:36:25.982Z] [2026-04-16T09:36:25.982Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T09:36:25.982Z] skipping: [an-a01n01] [2026-04-16T09:36:25.982Z] skipping: [an-a01n02] [2026-04-16T09:36:25.982Z] [2026-04-16T09:36:25.982Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T09:36:25.982Z] skipping: [an-a01n01] [2026-04-16T09:36:25.982Z] skipping: [an-a01n02] [2026-04-16T09:36:25.982Z] [2026-04-16T09:36:25.982Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T09:36:25.982Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:36:25.982Z] [2026-04-16T09:36:25.982Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:36:25.982Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:36:25.982Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:36:25.982Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:36:25.982Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:36:25.982Z] echo "pcs:" [2026-04-16T09:36:25.982Z] echo "$pcsstatus" [2026-04-16T09:36:25.982Z] exit 1 [2026-04-16T09:36:25.982Z] fi [2026-04-16T09:36:25.982Z] exit 0 [2026-04-16T09:36:25.982Z] ] *** [2026-04-16T09:36:28.448Z] ok: [an-a01n01] [2026-04-16T09:36:28.448Z] ok: [an-a01n02] [2026-04-16T09:36:28.448Z] [2026-04-16T09:36:28.448Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:36:28.448Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:36:28.448Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:36:28.448Z] ret=0 [2026-04-16T09:36:28.448Z] if [ "$servers" != "1" ]; then [2026-04-16T09:36:28.448Z] echo "Not all servers are running on the same node!" [2026-04-16T09:36:28.448Z] echo "$pcsstatus" [2026-04-16T09:36:28.448Z] ret=1 [2026-04-16T09:36:28.448Z] fi [2026-04-16T09:36:28.448Z] echo "$hypervisor" [2026-04-16T09:36:28.448Z] exit $ret [2026-04-16T09:36:28.448Z] ] *** [2026-04-16T09:36:30.275Z] ok: [an-a01n01] [2026-04-16T09:36:30.275Z] ok: [an-a01n02] [2026-04-16T09:36:30.275Z] [2026-04-16T09:36:30.275Z] PLAY [DR tests] **************************************************************** [2026-04-16T09:36:30.275Z] [2026-04-16T09:36:30.275Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:36:32.101Z] ok: [an-a01dr01] [2026-04-16T09:36:32.101Z] [2026-04-16T09:36:32.101Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured [2026-04-16T09:36:32.101Z] ] *** [2026-04-16T09:36:36.323Z] changed: [an-a01dr01] [2026-04-16T09:36:36.323Z] [2026-04-16T09:36:36.323Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g') [2026-04-16T09:36:36.323Z] echo "DB registered hostname: $drhostname" [2026-04-16T09:36:36.323Z] echo "current hostname: $(hostname)" [2026-04-16T09:36:36.323Z] if [ "$drhostname" != "$(hostname)" ]; then [2026-04-16T09:36:36.323Z] echo "DR hostnames are not matching" [2026-04-16T09:36:36.323Z] exit 1 [2026-04-16T09:36:36.323Z] fi [2026-04-16T09:36:36.323Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then [2026-04-16T09:36:36.323Z] echo "DR hostnames are not matching drnodes an-a01dr01" [2026-04-16T09:36:36.323Z] exit 1 [2026-04-16T09:36:36.323Z] fi [2026-04-16T09:36:36.323Z] ] *** [2026-04-16T09:36:40.397Z] changed: [an-a01dr01] [2026-04-16T09:36:40.397Z] [2026-04-16T09:36:40.397Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-04-16T09:36:40.397Z] echo "Detected a linked DR host!" [2026-04-16T09:36:40.397Z] exit 1 [2026-04-16T09:36:40.397Z] fi [2026-04-16T09:36:40.397Z] ] *** [2026-04-16T09:36:45.488Z] changed: [an-a01dr01] [2026-04-16T09:36:45.488Z] [2026-04-16T09:36:45.488Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01 [2026-04-16T09:36:45.488Z] ] *** [2026-04-16T09:36:50.575Z] changed: [an-a01dr01] [2026-04-16T09:36:50.575Z] [2026-04-16T09:36:50.575Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then [2026-04-16T09:36:50.575Z] echo "DR host failed to link to anvil" [2026-04-16T09:36:50.575Z] exit 1 [2026-04-16T09:36:50.575Z] fi [2026-04-16T09:36:50.575Z] ] *** [2026-04-16T09:36:55.670Z] changed: [an-a01dr01] [2026-04-16T09:36:55.670Z] [2026-04-16T09:36:55.670Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-04-16T09:36:55.670Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-04-16T09:36:55.670Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-04-16T09:36:55.670Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-04-16T09:36:55.670Z] exit 1 [2026-04-16T09:36:55.670Z] fi [2026-04-16T09:36:55.670Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then [2026-04-16T09:36:55.670Z] echo "Unable to link local vg uuid to storage group" [2026-04-16T09:36:55.670Z] exit 1 [2026-04-16T09:36:55.670Z] fi [2026-04-16T09:36:55.670Z] ] *** [2026-04-16T09:37:02.149Z] changed: [an-a01dr01] [2026-04-16T09:37:02.149Z] [2026-04-16T09:37:02.149Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then [2026-04-16T09:37:02.149Z] echo "Local vg does not appear to be linked to storage group" [2026-04-16T09:37:02.149Z] exit 1 [2026-04-16T09:37:02.149Z] fi [2026-04-16T09:37:02.149Z] ] *** [2026-04-16T09:37:06.221Z] changed: [an-a01dr01] [2026-04-16T09:37:06.221Z] [2026-04-16T09:37:06.221Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then [2026-04-16T09:37:06.221Z] echo "Unable to add protection to server an-test-deploy1" [2026-04-16T09:37:06.221Z] exit 1 [2026-04-16T09:37:06.221Z] fi [2026-04-16T09:37:06.221Z] ] *** [2026-04-16T09:37:11.308Z] changed: [an-a01dr01] [2026-04-16T09:37:11.308Z] [2026-04-16T09:37:11.308Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-04-16T09:37:11.308Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-16T09:37:11.308Z] [2026-04-16T09:37:11.308Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T09:37:11.308Z] exit 1 [2026-04-16T09:37:11.308Z] fi [2026-04-16T09:37:11.308Z] ] *** [2026-04-16T09:38:00.378Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-16T09:38:00.378Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-16T09:38:00.378Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-16T09:38:00.378Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left). [2026-04-16T09:38:00.378Z] ok: [an-a01dr01] [2026-04-16T09:38:00.378Z] [2026-04-16T09:38:00.378Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-04-16T09:38:00.378Z] short-throw) [2026-04-16T09:38:00.378Z] expected=2 [2026-04-16T09:38:00.378Z] protocol=A [2026-04-16T09:38:00.378Z] ;; [2026-04-16T09:38:00.378Z] sync) [2026-04-16T09:38:00.378Z] expected=3 [2026-04-16T09:38:00.378Z] protocol=C [2026-04-16T09:38:00.378Z] ;; [2026-04-16T09:38:00.378Z] *) [2026-04-16T09:38:00.378Z] echo "unknown DRBD protection method" [2026-04-16T09:38:00.378Z] exit 1 [2026-04-16T09:38:00.378Z] ;; [2026-04-16T09:38:00.378Z] esac [2026-04-16T09:38:00.378Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-04-16T09:38:00.378Z] echo "Unable to confirm correct DRBD protocol" [2026-04-16T09:38:00.378Z] exit 1 [2026-04-16T09:38:00.378Z] fi [2026-04-16T09:38:00.378Z] ] *** [2026-04-16T09:38:00.378Z] changed: [an-a01dr01] [2026-04-16T09:38:00.378Z] [2026-04-16T09:38:00.378Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:38:00.378Z] ] *** [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-16T09:39:54.670Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-16T09:39:54.670Z] ok: [an-a01dr01] [2026-04-16T09:39:54.670Z] [2026-04-16T09:39:54.670Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-04-16T09:39:54.670Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-16T09:39:54.670Z] [2026-04-16T09:39:54.670Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:39:54.670Z] ] *** [2026-04-16T09:39:54.670Z] ok: [an-a01dr01] [2026-04-16T09:39:54.670Z] [2026-04-16T09:39:54.670Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-04-16T09:39:54.670Z] echo "Unable to disconnect server from DR replication" [2026-04-16T09:39:54.670Z] exit 1 [2026-04-16T09:39:54.670Z] fi [2026-04-16T09:39:54.670Z] ] *** [2026-04-16T09:39:58.029Z] changed: [an-a01dr01] [2026-04-16T09:39:58.029Z] [2026-04-16T09:39:58.029Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:39:58.029Z] ] *** [2026-04-16T09:40:21.806Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T09:40:21.806Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T09:40:21.806Z] ok: [an-a01dr01] [2026-04-16T09:40:21.806Z] [2026-04-16T09:40:21.806Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-04-16T09:40:21.806Z] sleep 120 [2026-04-16T09:40:21.806Z] ] *** [2026-04-16T09:42:32.093Z] changed: [an-a01dr01 -> localhost] [2026-04-16T09:42:32.093Z] [2026-04-16T09:42:32.093Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-04-16T09:42:32.093Z] echo "Unable to update server DR replications" [2026-04-16T09:42:32.093Z] exit 1 [2026-04-16T09:42:32.093Z] fi [2026-04-16T09:42:32.093Z] ] *** [2026-04-16T09:42:32.093Z] changed: [an-a01dr01] [2026-04-16T09:42:32.093Z] [2026-04-16T09:42:32.093Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:42:32.093Z] ] *** [2026-04-16T09:43:06.193Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T09:43:06.193Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T09:43:06.193Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-04-16T09:43:06.193Z] ok: [an-a01dr01] [2026-04-16T09:43:06.193Z] [2026-04-16T09:43:06.193Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:43:06.193Z] ] *** [2026-04-16T09:43:26.431Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T09:43:26.431Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T09:43:26.431Z] ok: [an-a01dr01] [2026-04-16T09:43:26.431Z] [2026-04-16T09:43:26.431Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-04-16T09:43:26.431Z] echo "Unable to (re)connect server from DR replication" [2026-04-16T09:43:26.431Z] exit 1 [2026-04-16T09:43:26.431Z] fi [2026-04-16T09:43:26.431Z] ] *** [2026-04-16T09:43:31.519Z] changed: [an-a01dr01] [2026-04-16T09:43:31.519Z] [2026-04-16T09:43:31.519Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:43:31.519Z] ] *** [2026-04-16T09:44:05.621Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T09:44:05.621Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T09:44:05.621Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-04-16T09:44:05.621Z] ok: [an-a01dr01] [2026-04-16T09:44:05.621Z] [2026-04-16T09:44:05.621Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-04-16T09:44:05.621Z] echo "Unable to remove protection from server an-test-deploy1" [2026-04-16T09:44:05.621Z] exit 1 [2026-04-16T09:44:05.621Z] fi [2026-04-16T09:44:05.621Z] ] *** [2026-04-16T09:44:08.829Z] changed: [an-a01dr01] [2026-04-16T09:44:08.829Z] [2026-04-16T09:44:08.829Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-04-16T09:44:08.829Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-16T09:44:08.829Z] [2026-04-16T09:44:08.829Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:44:08.829Z] ] *** [2026-04-16T09:44:42.933Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-16T09:44:42.933Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-16T09:44:42.933Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-16T09:44:42.933Z] ok: [an-a01dr01] [2026-04-16T09:44:42.933Z] [2026-04-16T09:44:42.933Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:44:42.933Z] ] *** [2026-04-16T09:44:52.313Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed (60 retries left). [2026-04-16T09:44:52.313Z] ok: [an-a01dr01] [2026-04-16T09:44:52.313Z] [2026-04-16T09:44:52.313Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T09:44:52.313Z] exit 1 [2026-04-16T09:44:52.313Z] fi [2026-04-16T09:44:52.313Z] ] *** [2026-04-16T09:45:26.644Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-16T09:45:26.644Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-16T09:45:26.644Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-04-16T09:45:26.644Z] ok: [an-a01dr01] [2026-04-16T09:45:26.644Z] [2026-04-16T09:45:26.644Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-04-16T09:45:26.644Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-04-16T09:45:26.644Z] #exit 1 [2026-04-16T09:45:26.644Z] fi [2026-04-16T09:45:26.644Z] ] *** [2026-04-16T09:45:26.644Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-16T09:45:26.644Z] [2026-04-16T09:45:26.644Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then [2026-04-16T09:45:26.644Z] echo "Unable to add protection to server an-test-deploy1" [2026-04-16T09:45:26.644Z] exit 1 [2026-04-16T09:45:26.644Z] fi [2026-04-16T09:45:26.644Z] ] *** [2026-04-16T09:45:31.732Z] changed: [an-a01dr01] [2026-04-16T09:45:31.732Z] [2026-04-16T09:45:31.732Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-04-16T09:45:31.732Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-16T09:45:31.732Z] [2026-04-16T09:45:31.732Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T09:45:31.732Z] exit 1 [2026-04-16T09:45:31.732Z] fi [2026-04-16T09:45:31.732Z] ] *** [2026-04-16T09:46:06.102Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-16T09:46:06.102Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-16T09:46:06.102Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-16T09:46:06.102Z] ok: [an-a01dr01] [2026-04-16T09:46:06.102Z] [2026-04-16T09:46:06.102Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-04-16T09:46:06.102Z] short-throw) [2026-04-16T09:46:06.102Z] expected=2 [2026-04-16T09:46:06.102Z] protocol=A [2026-04-16T09:46:06.102Z] ;; [2026-04-16T09:46:06.102Z] sync) [2026-04-16T09:46:06.102Z] expected=3 [2026-04-16T09:46:06.102Z] protocol=C [2026-04-16T09:46:06.102Z] ;; [2026-04-16T09:46:06.102Z] *) [2026-04-16T09:46:06.102Z] echo "unknown DRBD protection method" [2026-04-16T09:46:06.102Z] exit 1 [2026-04-16T09:46:06.102Z] ;; [2026-04-16T09:46:06.102Z] esac [2026-04-16T09:46:06.102Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-04-16T09:46:06.102Z] echo "Unable to confirm correct DRBD protocol" [2026-04-16T09:46:06.102Z] exit 1 [2026-04-16T09:46:06.102Z] fi [2026-04-16T09:46:06.102Z] ] *** [2026-04-16T09:46:06.102Z] changed: [an-a01dr01] [2026-04-16T09:46:06.102Z] [2026-04-16T09:46:06.102Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:46:06.102Z] ] *** [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-16T09:48:15.491Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-16T09:48:15.491Z] ok: [an-a01dr01] [2026-04-16T09:48:15.491Z] [2026-04-16T09:48:15.491Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-04-16T09:48:15.491Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-16T09:48:15.491Z] [2026-04-16T09:48:15.491Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T09:48:15.491Z] ] *** [2026-04-16T09:48:15.491Z] ok: [an-a01dr01] [2026-04-16T09:48:15.491Z] [2026-04-16T09:48:15.491Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-04-16T09:48:15.491Z] echo "Unable to disconnect server from DR replication" [2026-04-16T09:48:15.491Z] exit 1 [2026-04-16T09:48:15.491Z] fi [2026-04-16T09:48:15.491Z] ] *** [2026-04-16T09:48:15.491Z] changed: [an-a01dr01] [2026-04-16T09:48:15.491Z] [2026-04-16T09:48:15.491Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:48:15.491Z] ] *** [2026-04-16T09:48:31.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T09:48:31.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T09:48:31.946Z] ok: [an-a01dr01] [2026-04-16T09:48:31.946Z] [2026-04-16T09:48:31.946Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-04-16T09:48:31.946Z] sleep 120 [2026-04-16T09:48:31.946Z] ] *** [2026-04-16T09:50:41.601Z] changed: [an-a01dr01 -> localhost] [2026-04-16T09:50:41.601Z] [2026-04-16T09:50:41.601Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-04-16T09:50:41.601Z] echo "Unable to update server DR replications" [2026-04-16T09:50:41.601Z] exit 1 [2026-04-16T09:50:41.601Z] fi [2026-04-16T09:50:41.601Z] ] *** [2026-04-16T09:50:41.601Z] changed: [an-a01dr01] [2026-04-16T09:50:41.601Z] [2026-04-16T09:50:41.601Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:50:41.601Z] ] *** [2026-04-16T09:51:15.703Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T09:51:15.703Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T09:51:15.703Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-04-16T09:51:15.703Z] ok: [an-a01dr01] [2026-04-16T09:51:15.703Z] [2026-04-16T09:51:15.703Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:51:15.703Z] ] *** [2026-04-16T09:51:39.436Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-16T09:51:39.436Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-16T09:51:39.436Z] ok: [an-a01dr01] [2026-04-16T09:51:39.436Z] [2026-04-16T09:51:39.436Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-04-16T09:51:39.436Z] echo "Unable to (re)connect server from DR replication" [2026-04-16T09:51:39.436Z] exit 1 [2026-04-16T09:51:39.436Z] fi [2026-04-16T09:51:39.436Z] ] *** [2026-04-16T09:51:40.774Z] changed: [an-a01dr01] [2026-04-16T09:51:40.774Z] [2026-04-16T09:51:40.774Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:51:40.774Z] ] *** [2026-04-16T09:52:04.514Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-16T09:52:04.514Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-16T09:52:04.514Z] ok: [an-a01dr01] [2026-04-16T09:52:04.514Z] [2026-04-16T09:52:04.515Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-04-16T09:52:04.515Z] echo "Unable to remove protection from server an-test-deploy1" [2026-04-16T09:52:04.515Z] exit 1 [2026-04-16T09:52:04.515Z] fi [2026-04-16T09:52:04.515Z] ] *** [2026-04-16T09:52:08.589Z] changed: [an-a01dr01] [2026-04-16T09:52:08.589Z] [2026-04-16T09:52:08.589Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-04-16T09:52:08.959Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-16T09:52:08.959Z] [2026-04-16T09:52:08.959Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T09:52:08.959Z] ] *** [2026-04-16T09:52:43.203Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-16T09:52:43.203Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-16T09:52:43.203Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-16T09:52:43.203Z] ok: [an-a01dr01] [2026-04-16T09:52:43.203Z] [2026-04-16T09:52:43.203Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T09:52:43.203Z] ] *** [2026-04-16T09:52:52.555Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed (60 retries left). [2026-04-16T09:52:52.555Z] ok: [an-a01dr01] [2026-04-16T09:52:52.555Z] [2026-04-16T09:52:52.555Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-04-16T09:52:52.555Z] exit 1 [2026-04-16T09:52:52.555Z] fi [2026-04-16T09:52:52.555Z] ] *** [2026-04-16T09:53:16.497Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-16T09:53:16.497Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-16T09:53:16.497Z] ok: [an-a01dr01] [2026-04-16T09:53:16.497Z] [2026-04-16T09:53:16.497Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-04-16T09:53:16.497Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-04-16T09:53:16.497Z] #exit 1 [2026-04-16T09:53:16.497Z] fi [2026-04-16T09:53:16.497Z] ] *** [2026-04-16T09:53:16.497Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-16T09:53:16.497Z] [2026-04-16T09:53:16.497Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-04-16T09:53:16.497Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-04-16T09:53:16.497Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-04-16T09:53:16.497Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-04-16T09:53:16.497Z] exit 1 [2026-04-16T09:53:16.497Z] fi [2026-04-16T09:53:16.497Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then [2026-04-16T09:53:16.497Z] echo "Unable to unlink local vg uuid to storage group" [2026-04-16T09:53:16.497Z] exit 1 [2026-04-16T09:53:16.497Z] fi [2026-04-16T09:53:16.497Z] ] *** [2026-04-16T09:53:22.778Z] changed: [an-a01dr01] [2026-04-16T09:53:22.778Z] [2026-04-16T09:53:22.778Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then [2026-04-16T09:53:22.778Z] echo "Local vg does not appear to be unlinked from storage group (#886)" [2026-04-16T09:53:22.778Z] exit 1 [2026-04-16T09:53:22.778Z] fi [2026-04-16T09:53:22.778Z] ] *** [2026-04-16T09:53:25.983Z] changed: [an-a01dr01] [2026-04-16T09:53:25.983Z] [2026-04-16T09:53:25.983Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01 [2026-04-16T09:53:25.983Z] ] *** [2026-04-16T09:53:31.069Z] changed: [an-a01dr01] [2026-04-16T09:53:31.069Z] [2026-04-16T09:53:31.069Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-04-16T09:53:31.069Z] echo "Detected a linked DR host!" [2026-04-16T09:53:31.069Z] exit 1 [2026-04-16T09:53:31.069Z] fi [2026-04-16T09:53:31.069Z] ] *** [2026-04-16T09:53:34.277Z] changed: [an-a01dr01] [2026-04-16T09:53:34.277Z] [2026-04-16T09:53:34.277Z] PLAY [upgrade server test] ***************************************************** [2026-04-16T09:53:34.277Z] [2026-04-16T09:53:34.277Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:53:36.741Z] ok: [an-a01n01] [2026-04-16T09:53:36.741Z] ok: [an-a01n02] [2026-04-16T09:53:36.741Z] [2026-04-16T09:53:36.741Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T09:53:36.741Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T09:53:36.741Z] [2026-04-16T09:53:36.741Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T09:53:36.741Z] ] *** [2026-04-16T09:53:40.096Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T09:53:40.096Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T09:53:40.096Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T09:53:40.096Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T09:53:40.096Z] [2026-04-16T09:53:40.096Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-04-16T09:53:40.096Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-16T09:53:40.096Z] [2026-04-16T09:53:40.096Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-04-16T09:53:40.096Z] if [ -z "$storage" ]; then [2026-04-16T09:53:40.096Z] echo "Unable to detect shared storage" [2026-04-16T09:53:40.096Z] exit 1 [2026-04-16T09:53:40.096Z] fi [2026-04-16T09:53:40.096Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:53:40.096Z] ] *** [2026-04-16T09:53:45.185Z] changed: [an-a01n02] [2026-04-16T09:53:45.185Z] changed: [an-a01n01] [2026-04-16T09:53:45.185Z] [2026-04-16T09:53:45.185Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-04-16T09:53:45.556Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-16T09:53:45.556Z] [2026-04-16T09:53:45.556Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-04-16T09:53:45.556Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-04-16T09:53:45.556Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-04-16T09:53:45.556Z] echo "Unable to detect all deploy iso" [2026-04-16T09:53:45.556Z] echo "Detected isos" [2026-04-16T09:53:45.556Z] echo "$isos" [2026-04-16T09:53:45.556Z] exit 1 [2026-04-16T09:53:45.556Z] fi [2026-04-16T09:53:45.556Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-04-16T09:53:45.556Z] if [ -z "$installiso" ]; then [2026-04-16T09:53:45.556Z] echo "Unable to detect install iso" [2026-04-16T09:53:45.556Z] echo "Detected isos" [2026-04-16T09:53:45.556Z] echo "$isos" [2026-04-16T09:53:45.556Z] exit 1 [2026-04-16T09:53:45.556Z] fi [2026-04-16T09:53:45.556Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-04-16T09:53:45.556Z] ] *** [2026-04-16T09:53:50.644Z] changed: [an-a01n01] [2026-04-16T09:53:50.644Z] changed: [an-a01n02] [2026-04-16T09:53:50.644Z] [2026-04-16T09:53:50.644Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-04-16T09:53:50.644Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-16T09:53:50.644Z] [2026-04-16T09:53:50.644Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T09:53:50.644Z] ] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] ok: [an-a01n01] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 852bd32f-0127-4aba-a37b-fc1c5d0eec27 _raw_params=anvil-provision-server -vv --log-secure \ [2026-04-16T09:53:52.472Z] --ci-test \ [2026-04-16T09:53:52.472Z] --name an-test-deploy{{ servernum }} \ [2026-04-16T09:53:52.472Z] --os almalinux9 \ [2026-04-16T09:53:52.472Z] --cpu 2 --ram 4G \ [2026-04-16T09:53:52.472Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-04-16T09:53:52.472Z] --install-media {{ installisouuid }} \ [2026-04-16T09:53:52.472Z] --driver-disc deploy{{ servernum }}.iso [2026-04-16T09:53:52.472Z] ] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n01] [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n01] [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n01] [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n01] [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-04-16T09:53:52.472Z] skipping: [an-a01n01] [2026-04-16T09:53:52.472Z] skipping: [an-a01n02] [2026-04-16T09:53:52.472Z] [2026-04-16T09:53:52.472Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-04-16T09:53:52.843Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-16T09:53:52.843Z] [2026-04-16T09:53:52.843Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:53:52.843Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-04-16T09:53:52.843Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-04-16T09:53:52.843Z] if [ "$numservers" != "$startedservers" ]; then [2026-04-16T09:53:52.843Z] echo "Detected $numservers but only $startedservers in Started state" [2026-04-16T09:53:52.843Z] echo "pcs:" [2026-04-16T09:53:52.843Z] echo "$pcsstatus" [2026-04-16T09:53:52.843Z] exit 1 [2026-04-16T09:53:52.843Z] fi [2026-04-16T09:53:52.843Z] exit 0 [2026-04-16T09:53:52.843Z] ] *** [2026-04-16T09:53:54.671Z] ok: [an-a01n01] [2026-04-16T09:53:54.671Z] ok: [an-a01n02] [2026-04-16T09:53:54.671Z] [2026-04-16T09:53:54.671Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-04-16T09:53:54.671Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-04-16T09:53:54.671Z] servers="$(echo "$hypervisor" | wc -l)" [2026-04-16T09:53:54.671Z] ret=0 [2026-04-16T09:53:54.671Z] if [ "$servers" != "1" ]; then [2026-04-16T09:53:54.671Z] echo "Not all servers are running on the same node!" [2026-04-16T09:53:54.671Z] echo "$pcsstatus" [2026-04-16T09:53:54.671Z] ret=1 [2026-04-16T09:53:54.671Z] fi [2026-04-16T09:53:54.671Z] echo "$hypervisor" [2026-04-16T09:53:54.671Z] exit $ret [2026-04-16T09:53:54.671Z] ] *** [2026-04-16T09:53:56.649Z] ok: [an-a01n02] [2026-04-16T09:53:56.649Z] ok: [an-a01n01] [2026-04-16T09:53:56.649Z] [2026-04-16T09:53:56.649Z] PLAY [enable upgrade repo] ***************************************************** [2026-04-16T09:53:56.649Z] [2026-04-16T09:53:56.649Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:53:59.954Z] ok: [an-a01n01] [2026-04-16T09:53:59.954Z] ok: [an-a01n02] [2026-04-16T09:53:59.954Z] ok: [an-a01dr01] [2026-04-16T09:53:59.954Z] ok: [an-striker02] [2026-04-16T09:53:59.954Z] ok: [an-striker01] [2026-04-16T09:53:59.954Z] [2026-04-16T09:53:59.954Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-04-16T09:53:59.954Z] dnf config-manager --set-enabled $updaterepo [2026-04-16T09:53:59.954Z] ] *** [2026-04-16T09:54:11.319Z] changed: [an-a01n01] [2026-04-16T09:54:11.319Z] changed: [an-a01n02] [2026-04-16T09:54:11.319Z] changed: [an-a01dr01] [2026-04-16T09:54:11.319Z] changed: [an-striker01] [2026-04-16T09:54:11.319Z] changed: [an-striker02] [2026-04-16T09:54:11.319Z] [2026-04-16T09:54:11.319Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-04-16T09:54:11.319Z] ok: [an-striker01] [2026-04-16T09:54:11.319Z] ok: [an-striker02] [2026-04-16T09:54:11.319Z] ok: [an-a01n01] [2026-04-16T09:54:11.319Z] ok: [an-a01n02] [2026-04-16T09:54:11.319Z] ok: [an-a01dr01] [2026-04-16T09:54:11.319Z] [2026-04-16T09:54:11.319Z] PLAY [start upgrade process] *************************************************** [2026-04-16T09:54:11.319Z] [2026-04-16T09:54:11.319Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T09:54:13.785Z] ok: [an-striker01] [2026-04-16T09:54:13.785Z] [2026-04-16T09:54:13.785Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-04-16T09:54:13.785Z] ] *** [2026-04-16T10:22:27.301Z] changed: [an-striker01] [2026-04-16T10:22:27.301Z] [2026-04-16T10:22:27.301Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-04-16T10:22:55.764Z] changed: [an-striker01] [2026-04-16T10:22:55.764Z] [2026-04-16T10:22:55.764Z] PLAY [reconnect all nodes] ***************************************************** [2026-04-16T10:22:55.764Z] [2026-04-16T10:22:55.764Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:22:56.905Z] ok: [an-a01n02] [2026-04-16T10:22:56.905Z] ok: [an-a01n01] [2026-04-16T10:22:56.905Z] ok: [an-a01dr01] [2026-04-16T10:22:56.905Z] ok: [an-striker02] [2026-04-16T10:22:56.905Z] ok: [an-striker01] [2026-04-16T10:22:56.905Z] [2026-04-16T10:22:56.905Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-04-16T10:22:56.905Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01 [2026-04-16T10:22:56.905Z] [2026-04-16T10:22:56.905Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-04-16T10:26:21.954Z] ok: [an-striker01 -> localhost] [2026-04-16T10:26:21.954Z] ok: [an-striker02 -> localhost] [2026-04-16T10:26:21.954Z] ok: [an-a01n01 -> localhost] [2026-04-16T10:26:21.954Z] ok: [an-a01n02 -> localhost] [2026-04-16T10:26:21.954Z] ok: [an-a01dr01 -> localhost] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-04-16T10:26:21.954Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-04-16T10:26:21.954Z] exit 1 [2026-04-16T10:26:21.954Z] fi [2026-04-16T10:26:21.954Z] ] *** [2026-04-16T10:26:21.954Z] changed: [an-a01n02] [2026-04-16T10:26:21.954Z] changed: [an-striker01] [2026-04-16T10:26:21.954Z] changed: [an-a01n01] [2026-04-16T10:26:21.954Z] changed: [an-a01dr01] [2026-04-16T10:26:21.954Z] changed: [an-striker02] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] PLAY [check all nodes status after upgrade] ************************************ [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:26:21.954Z] ok: [an-a01n02] [2026-04-16T10:26:21.954Z] ok: [an-a01n01] [2026-04-16T10:26:21.954Z] ok: [an-a01dr01] [2026-04-16T10:26:21.954Z] ok: [an-striker01] [2026-04-16T10:26:21.954Z] ok: [an-striker02] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-04-16T10:26:21.954Z] echo "perl-Log-Journald update not found!" [2026-04-16T10:26:21.954Z] exit 1 [2026-04-16T10:26:21.954Z] fi [2026-04-16T10:26:21.954Z] ] *** [2026-04-16T10:26:21.954Z] changed: [an-a01n01] [2026-04-16T10:26:21.954Z] changed: [an-a01n02] [2026-04-16T10:26:21.954Z] changed: [an-a01dr01] [2026-04-16T10:26:21.954Z] changed: [an-striker02] [2026-04-16T10:26:21.954Z] changed: [an-striker01] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] PLAY [check nodes status after upgrade] **************************************** [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:26:21.954Z] ok: [an-a01n02] [2026-04-16T10:26:21.954Z] ok: [an-a01dr01] [2026-04-16T10:26:21.954Z] ok: [an-a01n01] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-04-16T10:26:21.954Z] echo "akmod-drbd update not found!" [2026-04-16T10:26:21.954Z] exit 1 [2026-04-16T10:26:21.954Z] fi [2026-04-16T10:26:21.954Z] ] *** [2026-04-16T10:26:21.954Z] changed: [an-a01n02] [2026-04-16T10:26:21.954Z] changed: [an-a01dr01] [2026-04-16T10:26:21.954Z] changed: [an-a01n01] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] PLAY [check hosted node status after upgrade] ********************************** [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:26:21.954Z] ok: [an-a01n01] [2026-04-16T10:26:21.954Z] ok: [an-a01n02] [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T10:26:21.954Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T10:26:21.954Z] [2026-04-16T10:26:21.954Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T10:26:21.954Z] ] *** [2026-04-16T10:26:24.171Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T10:26:24.171Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T10:26:24.171Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T10:26:24.171Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T10:26:24.171Z] [2026-04-16T10:26:24.171Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-04-16T10:26:24.171Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-04-16T10:26:24.171Z] [2026-04-16T10:26:24.171Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-04-16T10:26:24.171Z] ] *** [2026-04-16T10:26:25.998Z] ok: [an-a01n02] [2026-04-16T10:26:25.998Z] ok: [an-a01n01] [2026-04-16T10:26:25.998Z] [2026-04-16T10:26:25.998Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-04-16T10:26:25.998Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-04-16T10:26:25.999Z] [2026-04-16T10:26:25.999Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-04-16T10:26:25.999Z] ] *** [2026-04-16T10:26:26.374Z] ok: [an-a01n02] [2026-04-16T10:26:26.374Z] ok: [an-a01n01] [2026-04-16T10:26:26.374Z] [2026-04-16T10:26:26.374Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-04-16T10:26:26.374Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-04-16T10:26:26.374Z] [2026-04-16T10:26:26.374Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-04-16T10:26:26.374Z] ] *** [2026-04-16T10:26:27.166Z] ok: [an-a01n01] [2026-04-16T10:26:27.166Z] ok: [an-a01n02] [2026-04-16T10:26:27.166Z] [2026-04-16T10:26:27.166Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-04-16T10:26:27.166Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-04-16T10:26:27.166Z] [2026-04-16T10:26:27.166Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-04-16T10:26:27.166Z] ] *** [2026-04-16T10:26:28.439Z] ok: [an-a01n01] [2026-04-16T10:26:28.439Z] ok: [an-a01n02] [2026-04-16T10:26:28.439Z] [2026-04-16T10:26:28.439Z] PLAY [test anvil package self update] ****************************************** [2026-04-16T10:26:28.439Z] [2026-04-16T10:26:28.439Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:26:31.717Z] ok: [an-a01n02] [2026-04-16T10:26:31.717Z] ok: [an-a01n01] [2026-04-16T10:26:31.717Z] ok: [an-a01dr01] [2026-04-16T10:26:31.717Z] ok: [an-striker02] [2026-04-16T10:26:31.717Z] ok: [an-striker01] [2026-04-16T10:26:31.717Z] [2026-04-16T10:26:31.717Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-04-16T10:26:34.923Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-16T10:26:34.923Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-16T10:26:34.923Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-16T10:26:34.923Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-16T10:26:34.923Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-16T10:26:34.923Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-16T10:26:34.923Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-16T10:26:34.923Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-16T10:26:34.923Z] changed: [an-striker02] => (item=scancore.service) [2026-04-16T10:26:34.923Z] changed: [an-striker01] => (item=scancore.service) [2026-04-16T10:26:34.923Z] [2026-04-16T10:26:34.923Z] PLAY [test striker package self update] **************************************** [2026-04-16T10:26:34.923Z] [2026-04-16T10:26:34.923Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:26:36.248Z] ok: [an-striker01] [2026-04-16T10:26:36.248Z] ok: [an-striker02] [2026-04-16T10:26:36.248Z] [2026-04-16T10:26:36.248Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-04-16T10:26:36.248Z] ] *** [2026-04-16T10:27:01.013Z] changed: [an-striker01] [2026-04-16T10:27:01.013Z] changed: [an-striker02] [2026-04-16T10:27:01.013Z] [2026-04-16T10:27:01.013Z] PLAY [test node package self update] ******************************************* [2026-04-16T10:27:01.013Z] [2026-04-16T10:27:01.013Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:27:01.869Z] ok: [an-a01n01] [2026-04-16T10:27:01.869Z] ok: [an-a01n02] [2026-04-16T10:27:01.869Z] [2026-04-16T10:27:01.869Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-04-16T10:27:01.869Z] ] *** [2026-04-16T10:27:25.699Z] changed: [an-a01n02] [2026-04-16T10:27:25.699Z] changed: [an-a01n01] [2026-04-16T10:27:25.699Z] [2026-04-16T10:27:25.699Z] PLAY [test dr package self update] ********************************************* [2026-04-16T10:27:25.699Z] [2026-04-16T10:27:25.699Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:27:26.971Z] ok: [an-a01dr01] [2026-04-16T10:27:26.971Z] [2026-04-16T10:27:26.971Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr [2026-04-16T10:27:26.971Z] ] *** [2026-04-16T10:27:51.734Z] changed: [an-a01dr01] [2026-04-16T10:27:51.734Z] [2026-04-16T10:27:51.734Z] PLAY [test anvil package self update] ****************************************** [2026-04-16T10:27:51.734Z] [2026-04-16T10:27:51.734Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:27:52.531Z] ok: [an-striker01] [2026-04-16T10:27:52.531Z] ok: [an-a01n02] [2026-04-16T10:27:52.531Z] ok: [an-striker02] [2026-04-16T10:27:52.531Z] ok: [an-a01dr01] [2026-04-16T10:27:52.531Z] ok: [an-a01n01] [2026-04-16T10:27:52.531Z] [2026-04-16T10:27:52.531Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-04-16T10:27:55.395Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-16T10:27:55.395Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-16T10:27:55.395Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-16T10:27:55.395Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-16T10:27:55.395Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-16T10:27:55.395Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-16T10:27:55.395Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-16T10:27:55.395Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-16T10:27:55.395Z] changed: [an-striker01] => (item=scancore.service) [2026-04-16T10:27:55.395Z] changed: [an-striker02] => (item=scancore.service) [2026-04-16T10:27:55.395Z] [2026-04-16T10:27:55.395Z] PLAY [delete server tests] ***************************************************** [2026-04-16T10:27:55.395Z] [2026-04-16T10:27:55.395Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:27:57.226Z] ok: [an-a01n02] [2026-04-16T10:27:57.226Z] ok: [an-a01n01] [2026-04-16T10:27:57.226Z] [2026-04-16T10:27:57.226Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-04-16T10:27:57.226Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-16T10:27:57.226Z] [2026-04-16T10:27:57.226Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-04-16T10:27:57.226Z] ] *** [2026-04-16T10:28:01.304Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-16T10:28:01.304Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-16T10:28:01.304Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-16T10:28:01.304Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-16T10:28:01.304Z] [2026-04-16T10:28:01.304Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-04-16T10:28:01.304Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-16T10:28:01.304Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-16T10:28:01.304Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-16T10:28:01.304Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-16T10:28:01.304Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-16T10:28:01.304Z] [2026-04-16T10:28:01.304Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:28:01.304Z] ] *** [2026-04-16T10:28:02.577Z] skipping: [an-a01n01] [2026-04-16T10:28:02.577Z] ok: [an-a01n02] [2026-04-16T10:28:02.577Z] [2026-04-16T10:28:02.577Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T10:28:02.577Z] --force \ [2026-04-16T10:28:02.577Z] --server an-test-deploy{{ servernum }} [2026-04-16T10:28:02.577Z] ] *** [2026-04-16T10:28:10.666Z] skipping: [an-a01n01] [2026-04-16T10:28:10.666Z] changed: [an-a01n02] [2026-04-16T10:28:10.666Z] [2026-04-16T10:28:10.666Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:28:10.666Z] ] *** [2026-04-16T10:28:59.614Z] skipping: [an-a01n01] [2026-04-16T10:28:59.614Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-04-16T10:28:59.614Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-04-16T10:28:59.614Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-04-16T10:28:59.614Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-04-16T10:28:59.614Z] ok: [an-a01n02] [2026-04-16T10:28:59.614Z] [2026-04-16T10:28:59.614Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T10:28:59.614Z] ] *** [2026-04-16T10:28:59.614Z] skipping: [an-a01n01] [2026-04-16T10:28:59.614Z] ok: [an-a01n02] [2026-04-16T10:28:59.614Z] [2026-04-16T10:28:59.614Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T10:28:59.614Z] ] *** [2026-04-16T10:28:59.614Z] skipping: [an-a01n01] [2026-04-16T10:28:59.614Z] ok: [an-a01n02] [2026-04-16T10:28:59.614Z] [2026-04-16T10:28:59.614Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:28:59.614Z] ] *** [2026-04-16T10:29:01.442Z] skipping: [an-a01n01] [2026-04-16T10:29:01.442Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.990628", "end": "2026-04-16 06:29:00.966656", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 06:28:59.976028", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T10:29:01.442Z] ...ignoring [2026-04-16T10:29:01.442Z] [2026-04-16T10:29:01.442Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T10:29:01.442Z] --force \ [2026-04-16T10:29:01.442Z] --server an-test-deploy{{ servernum }} [2026-04-16T10:29:01.442Z] ] *** [2026-04-16T10:29:01.442Z] skipping: [an-a01n01] [2026-04-16T10:29:01.442Z] skipping: [an-a01n02] [2026-04-16T10:29:01.442Z] [2026-04-16T10:29:01.442Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:01.442Z] ] *** [2026-04-16T10:29:01.442Z] skipping: [an-a01n01] [2026-04-16T10:29:01.442Z] skipping: [an-a01n02] [2026-04-16T10:29:01.442Z] [2026-04-16T10:29:01.442Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T10:29:01.442Z] ] *** [2026-04-16T10:29:01.442Z] skipping: [an-a01n01] [2026-04-16T10:29:01.442Z] skipping: [an-a01n02] [2026-04-16T10:29:01.442Z] [2026-04-16T10:29:01.442Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T10:29:01.442Z] ] *** [2026-04-16T10:29:01.442Z] skipping: [an-a01n01] [2026-04-16T10:29:01.442Z] skipping: [an-a01n02] [2026-04-16T10:29:01.442Z] [2026-04-16T10:29:01.442Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:01.442Z] ] *** [2026-04-16T10:29:03.437Z] skipping: [an-a01n01] [2026-04-16T10:29:03.437Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.896350", "end": "2026-04-16 06:29:02.864298", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 06:29:01.967948", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T10:29:03.437Z] ...ignoring [2026-04-16T10:29:03.437Z] [2026-04-16T10:29:03.437Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T10:29:03.437Z] --force \ [2026-04-16T10:29:03.437Z] --server an-test-deploy{{ servernum }} [2026-04-16T10:29:03.437Z] ] *** [2026-04-16T10:29:03.437Z] skipping: [an-a01n01] [2026-04-16T10:29:03.437Z] skipping: [an-a01n02] [2026-04-16T10:29:03.437Z] [2026-04-16T10:29:03.437Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:03.437Z] ] *** [2026-04-16T10:29:03.437Z] skipping: [an-a01n01] [2026-04-16T10:29:03.437Z] skipping: [an-a01n02] [2026-04-16T10:29:03.437Z] [2026-04-16T10:29:03.437Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T10:29:03.437Z] ] *** [2026-04-16T10:29:03.437Z] skipping: [an-a01n01] [2026-04-16T10:29:03.437Z] skipping: [an-a01n02] [2026-04-16T10:29:03.437Z] [2026-04-16T10:29:03.437Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T10:29:03.437Z] ] *** [2026-04-16T10:29:03.437Z] skipping: [an-a01n01] [2026-04-16T10:29:03.437Z] skipping: [an-a01n02] [2026-04-16T10:29:03.437Z] [2026-04-16T10:29:03.437Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:03.437Z] ] *** [2026-04-16T10:29:06.213Z] skipping: [an-a01n01] [2026-04-16T10:29:06.213Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.918834", "end": "2026-04-16 06:29:04.704496", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 06:29:03.785662", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T10:29:06.213Z] ...ignoring [2026-04-16T10:29:06.213Z] [2026-04-16T10:29:06.213Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T10:29:06.213Z] --force \ [2026-04-16T10:29:06.213Z] --server an-test-deploy{{ servernum }} [2026-04-16T10:29:06.213Z] ] *** [2026-04-16T10:29:06.213Z] skipping: [an-a01n01] [2026-04-16T10:29:06.213Z] skipping: [an-a01n02] [2026-04-16T10:29:06.213Z] [2026-04-16T10:29:06.213Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:06.213Z] ] *** [2026-04-16T10:29:06.213Z] skipping: [an-a01n01] [2026-04-16T10:29:06.213Z] skipping: [an-a01n02] [2026-04-16T10:29:06.213Z] [2026-04-16T10:29:06.213Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T10:29:06.213Z] ] *** [2026-04-16T10:29:06.213Z] skipping: [an-a01n01] [2026-04-16T10:29:06.213Z] skipping: [an-a01n02] [2026-04-16T10:29:06.213Z] [2026-04-16T10:29:06.213Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T10:29:06.213Z] ] *** [2026-04-16T10:29:06.213Z] skipping: [an-a01n01] [2026-04-16T10:29:06.213Z] skipping: [an-a01n02] [2026-04-16T10:29:06.213Z] [2026-04-16T10:29:06.213Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:06.213Z] ] *** [2026-04-16T10:29:06.586Z] skipping: [an-a01n01] [2026-04-16T10:29:06.586Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.838193", "end": "2026-04-16 06:29:06.450734", "msg": "non-zero return code", "rc": 1, "start": "2026-04-16 06:29:05.612541", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-16T10:29:06.586Z] ...ignoring [2026-04-16T10:29:06.586Z] [2026-04-16T10:29:06.586Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-04-16T10:29:06.586Z] --force \ [2026-04-16T10:29:06.586Z] --server an-test-deploy{{ servernum }} [2026-04-16T10:29:06.586Z] ] *** [2026-04-16T10:29:06.958Z] skipping: [an-a01n01] [2026-04-16T10:29:06.958Z] skipping: [an-a01n02] [2026-04-16T10:29:06.958Z] [2026-04-16T10:29:06.958Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-04-16T10:29:06.958Z] ] *** [2026-04-16T10:29:06.958Z] skipping: [an-a01n01] [2026-04-16T10:29:06.958Z] skipping: [an-a01n02] [2026-04-16T10:29:06.958Z] [2026-04-16T10:29:06.958Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-04-16T10:29:06.958Z] ] *** [2026-04-16T10:29:06.958Z] skipping: [an-a01n01] [2026-04-16T10:29:06.958Z] skipping: [an-a01n02] [2026-04-16T10:29:06.958Z] [2026-04-16T10:29:06.958Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-04-16T10:29:06.958Z] ] *** [2026-04-16T10:29:06.958Z] skipping: [an-a01n01] [2026-04-16T10:29:06.958Z] skipping: [an-a01n02] [2026-04-16T10:29:06.958Z] [2026-04-16T10:29:06.958Z] PLAY [Check anvil available resources from striker01] ************************** [2026-04-16T10:29:06.958Z] [2026-04-16T10:29:06.958Z] TASK [Gathering Facts ] ******************************************************** [2026-04-16T10:29:09.820Z] ok: [an-striker01] [2026-04-16T10:29:09.820Z] [2026-04-16T10:29:09.820Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-04-16T10:29:09.820Z] sleep 300 [2026-04-16T10:29:09.820Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-04-16T10:29:09.820Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-04-16T10:29:09.820Z] ] *** [2026-04-16T10:34:23.932Z] ok: [an-striker01] [2026-04-16T10:34:23.932Z] [2026-04-16T10:34:23.932Z] PLAY RECAP ********************************************************************* [2026-04-16T10:34:23.932Z] an-a01dr01 : ok=90 changed=38 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [2026-04-16T10:34:23.932Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-04-16T10:34:23.932Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-04-16T10:34:23.932Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-16T10:34:23.932Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-16T10:34:23.932Z] [2026-04-16T10:34:23.932Z] tar: Removing leading `/' from member names [2026-04-16T10:34:23.932Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-16T10:34:23.932Z] /root/ci-tools [2026-04-16T10:34:23.932Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/348.tar.gz [2026-04-16T10:34:23.932Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-04-16T10:34:23.932Z] This might take a few seconds. [2026-04-16T10:34:23.932Z] [2026-04-16T10:34:23.932Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T10:34:23.932Z] 𐄂 Disconnected from Red Hat Insights [2026-04-16T10:34:23.932Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-04-16T10:34:23.932Z] [2026-04-16T10:34:23.932Z] Manage your connected systems: https://red.ht/connector [2026-04-16T10:34:30.219Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T10:34:30.219Z] [2026-04-16T10:34:30.219Z] 1. Register with subscription-manager [2026-04-16T10:34:30.219Z] # subscription-manager register [2026-04-16T10:34:30.219Z] [2026-04-16T10:34:30.219Z] 2. Register with insights-client [2026-04-16T10:34:30.219Z] # insights-client --register [2026-04-16T10:34:31.013Z] This system is currently not registered. [2026-04-16T10:34:31.387Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-04-16T10:34:31.387Z] This might take a few seconds. [2026-04-16T10:34:31.387Z] [2026-04-16T10:34:31.387Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T10:34:39.085Z] 𐄂 Disconnected from Red Hat Insights [2026-04-16T10:34:39.085Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-04-16T10:34:39.085Z] [2026-04-16T10:34:39.085Z] Manage your connected systems: https://red.ht/connector [2026-04-16T10:34:45.407Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T10:34:45.407Z] [2026-04-16T10:34:45.407Z] 1. Register with subscription-manager [2026-04-16T10:34:45.407Z] # subscription-manager register [2026-04-16T10:34:45.407Z] [2026-04-16T10:34:45.407Z] 2. Register with insights-client [2026-04-16T10:34:45.407Z] # insights-client --register [2026-04-16T10:34:45.778Z] This system is currently not registered. [2026-04-16T10:34:46.149Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-04-16T10:34:46.149Z] This might take a few seconds. [2026-04-16T10:34:46.149Z] [2026-04-16T10:34:46.149Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T10:34:51.245Z] 𐄂 Disconnected from Red Hat Insights [2026-04-16T10:34:52.039Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-04-16T10:34:52.039Z] [2026-04-16T10:34:52.039Z] Manage your connected systems: https://red.ht/connector [2026-04-16T10:34:57.147Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T10:34:57.147Z] [2026-04-16T10:34:57.147Z] 1. Register with subscription-manager [2026-04-16T10:34:57.147Z] # subscription-manager register [2026-04-16T10:34:57.147Z] [2026-04-16T10:34:57.147Z] 2. Register with insights-client [2026-04-16T10:34:57.147Z] # insights-client --register [2026-04-16T10:34:57.939Z] This system is currently not registered. [2026-04-16T10:34:58.315Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-04-16T10:34:58.315Z] This might take a few seconds. [2026-04-16T10:34:58.315Z] [2026-04-16T10:34:58.315Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T10:35:03.797Z] 𐄂 Disconnected from Red Hat Insights [2026-04-16T10:35:04.589Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-04-16T10:35:04.589Z] [2026-04-16T10:35:04.589Z] Manage your connected systems: https://red.ht/connector [2026-04-16T10:35:08.699Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T10:35:08.699Z] [2026-04-16T10:35:08.699Z] 1. Register with subscription-manager [2026-04-16T10:35:08.699Z] # subscription-manager register [2026-04-16T10:35:08.699Z] [2026-04-16T10:35:08.699Z] 2. Register with insights-client [2026-04-16T10:35:08.699Z] # insights-client --register [2026-04-16T10:35:09.615Z] This system is currently not registered. [2026-04-16T10:35:10.408Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-04-16T10:35:10.408Z] This might take a few seconds. [2026-04-16T10:35:10.408Z] [2026-04-16T10:35:10.408Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-16T10:35:15.617Z] 𐄂 Disconnected from Red Hat Insights [2026-04-16T10:35:16.421Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-04-16T10:35:16.421Z] [2026-04-16T10:35:16.421Z] Manage your connected systems: https://red.ht/connector [2026-04-16T10:35:21.673Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-16T10:35:21.673Z] [2026-04-16T10:35:21.673Z] 1. Register with subscription-manager [2026-04-16T10:35:21.673Z] # subscription-manager register [2026-04-16T10:35:21.673Z] [2026-04-16T10:35:21.673Z] 2. Register with insights-client [2026-04-16T10:35:21.673Z] # insights-client --register [2026-04-16T10:35:21.673Z] This system is currently not registered. [2026-04-16T10:35:21.768Z] [Pipeline] } [2026-04-16T10:35:21.771Z] [Pipeline] // timeout [2026-04-16T10:35:21.774Z] [Pipeline] } [2026-04-16T10:35:21.776Z] [Pipeline] // script [2026-04-16T10:35:21.778Z] [Pipeline] } [2026-04-16T10:35:21.780Z] [Pipeline] // stage [2026-04-16T10:35:21.783Z] [Pipeline] stage [2026-04-16T10:35:21.783Z] [Pipeline] { (Declarative: Post Actions) [2026-04-16T10:35:21.789Z] [Pipeline] cleanWs [2026-04-16T10:35:21.865Z] [WS-CLEANUP] Deleting project workspace... [2026-04-16T10:35:21.865Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-16T10:35:22.017Z] [WS-CLEANUP] done [2026-04-16T10:35:22.020Z] [Pipeline] script [2026-04-16T10:35:22.021Z] [Pipeline] { [2026-04-16T10:35:22.035Z] [Pipeline] stage [2026-04-16T10:35:22.036Z] [Pipeline] { (Copying logs) [2026-04-16T10:35:22.069Z] [Pipeline] node [2026-04-16T10:35:22.077Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-04-16T10:35:22.078Z] [Pipeline] { [2026-04-16T10:35:22.111Z] [Pipeline] node [2026-04-16T10:35:22.119Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-04-16T10:35:22.120Z] [Pipeline] { [2026-04-16T10:35:22.130Z] [Pipeline] sh [2026-04-16T10:35:22.385Z] + mkdir -p /var/lib/jenkins/locks [2026-04-16T10:35:22.394Z] [Pipeline] echo [2026-04-16T10:35:22.394Z] RWLock: FD for lock log_archive in stage projectFinishUp is 471 [2026-04-16T10:35:22.397Z] [Pipeline] sleep [2026-04-16T10:35:22.398Z] No need to sleep any longer [2026-04-16T10:35:22.400Z] [Pipeline] echo [2026-04-16T10:35:22.400Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-04-16T10:35:22.403Z] [Pipeline] } [2026-04-16T10:35:22.406Z] [Pipeline] // node [2026-04-16T10:35:22.409Z] [Pipeline] echo [2026-04-16T10:35:22.409Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/348/ [2026-04-16T10:35:22.410Z] [Pipeline] sh [2026-04-16T10:35:22.665Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/348/ [2026-04-16T10:35:22.666Z] [Pipeline] sh