[2026-05-21T04:55:34.976Z] Started by upstream project "anvil/anvil-pipeline/main" build number 292 [2026-05-21T04:55:34.976Z] originally caused by: [2026-05-21T04:55:34.976Z] Push event to branch main [2026-05-21T04:55:35.893Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-05-21T04:55:35.903Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-05-21T04:55:36.116Z] Examining kronosnet/ci-tools [2026-05-21T04:55:36.116Z] Attempting to resolve main as a branch [2026-05-21T04:55:36.263Z] Resolved main as branch main at revision 31336d47b51e45ff43c196188035fba09b843a38 [2026-05-21T04:55:36.264Z] Selected Git installation does not exist. Using Default [2026-05-21T04:55:36.264Z] The recommended git tool is: NONE [2026-05-21T04:55:36.264Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-05-21T04:55:36.267Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-05-21T04:55:36.267Z] Wiping out workspace first. [2026-05-21T04:55:36.285Z] Cloning the remote Git repository [2026-05-21T04:55:36.285Z] Using shallow clone with depth 1 [2026-05-21T04:55:36.285Z] Avoid fetching tags [2026-05-21T04:55:36.285Z] Honoring refspec on initial clone [2026-05-21T04:55:36.285Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-05-21T04:55:36.285Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-05-21T04:55:36.295Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-05-21T04:55:36.295Z] > git --version # timeout=10 [2026-05-21T04:55:36.302Z] > git --version # 'git version 2.47.3' [2026-05-21T04:55:36.302Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-05-21T04:55:36.308Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-21T04:55:36.509Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-05-21T04:55:36.516Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-21T04:55:36.524Z] Avoid second fetch [2026-05-21T04:55:36.524Z] Checking out Revision 31336d47b51e45ff43c196188035fba09b843a38 (main) [2026-05-21T04:55:36.524Z] > git config core.sparsecheckout # timeout=10 [2026-05-21T04:55:36.531Z] > git checkout -f 31336d47b51e45ff43c196188035fba09b843a38 # timeout=10 [2026-05-21T04:55:36.562Z] Commit message: "kronosnet: main branch can do parallel testing" [2026-05-21T04:55:36.570Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-05-21T04:55:36.733Z] Examining kronosnet/ci-tools [2026-05-21T04:55:36.733Z] Attempting to resolve main as a branch [2026-05-21T04:55:36.884Z] Resolved main as branch main at revision 31336d47b51e45ff43c196188035fba09b843a38 [2026-05-21T04:55:36.885Z] Selected Git installation does not exist. Using Default [2026-05-21T04:55:36.885Z] The recommended git tool is: NONE [2026-05-21T04:55:36.885Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-05-21T04:55:36.888Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-05-21T04:55:36.888Z] Wiping out workspace first. [2026-05-21T04:55:36.915Z] Cloning the remote Git repository [2026-05-21T04:55:36.915Z] Using shallow clone with depth 1 [2026-05-21T04:55:36.915Z] Avoid fetching tags [2026-05-21T04:55:36.915Z] Honoring refspec on initial clone [2026-05-21T04:55:36.915Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-05-21T04:55:36.915Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-05-21T04:55:36.929Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-05-21T04:55:36.929Z] > git --version # timeout=10 [2026-05-21T04:55:36.936Z] > git --version # 'git version 2.47.3' [2026-05-21T04:55:36.936Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-05-21T04:55:36.942Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-21T04:55:37.130Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-05-21T04:55:37.138Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-21T04:55:37.146Z] Avoid second fetch [2026-05-21T04:55:37.146Z] Checking out Revision 31336d47b51e45ff43c196188035fba09b843a38 (main) [2026-05-21T04:55:37.146Z] > git config core.sparsecheckout # timeout=10 [2026-05-21T04:55:37.156Z] > git checkout -f 31336d47b51e45ff43c196188035fba09b843a38 # timeout=10 [2026-05-21T04:55:37.194Z] Commit message: "kronosnet: main branch can do parallel testing" [2026-05-21T04:55:37.259Z] [Pipeline] Start of Pipeline [2026-05-21T04:55:37.356Z] [Pipeline] node [2026-05-21T04:55:37.358Z] Running on anvil-ci-bm-phy04 in /srv/workspace/anvil/anvil-build-functional-testing [2026-05-21T04:55:37.359Z] [Pipeline] { [2026-05-21T04:55:37.366Z] [Pipeline] stage [2026-05-21T04:55:37.367Z] [Pipeline] { (Prep build env) [2026-05-21T04:55:37.376Z] [Pipeline] script [2026-05-21T04:55:37.378Z] [Pipeline] { [2026-05-21T04:55:37.381Z] [Pipeline] cleanWs [2026-05-21T04:55:37.432Z] [WS-CLEANUP] Deleting project workspace... [2026-05-21T04:55:37.432Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-05-21T04:55:37.457Z] [WS-CLEANUP] done [2026-05-21T04:55:37.476Z] [Pipeline] lock [2026-05-21T04:55:37.476Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-05-21T04:55:37.477Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-05-21T04:55:37.477Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-05-21T04:55:37.477Z] [Pipeline] { [2026-05-21T04:55:37.480Z] [Pipeline] } [2026-05-21T04:55:37.480Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-05-21T04:55:37.482Z] [Pipeline] // lock [2026-05-21T04:55:37.487Z] [Pipeline] } [2026-05-21T04:55:37.489Z] [Pipeline] // script [2026-05-21T04:55:37.491Z] [Pipeline] } [2026-05-21T04:55:37.493Z] [Pipeline] // stage [2026-05-21T04:55:37.495Z] [Pipeline] stage [2026-05-21T04:55:37.495Z] [Pipeline] { (Run functional testing) [2026-05-21T04:55:37.501Z] [Pipeline] script [2026-05-21T04:55:37.503Z] [Pipeline] { [2026-05-21T04:55:37.522Z] [Pipeline] echo [2026-05-21T04:55:37.522Z] runWithTimeout 360 [2026-05-21T04:55:37.523Z] [Pipeline] timeout [2026-05-21T04:55:37.523Z] Timeout set to expire in 6 hr 0 min [2026-05-21T04:55:37.524Z] [Pipeline] { [2026-05-21T04:55:37.528Z] [Pipeline] sh [2026-05-21T04:55:38.053Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 ci ci all [2026-05-21T04:55:38.053Z] CITBRANCH: main [2026-05-21T04:55:38.053Z] CITHOME: /root/ci-tools [2026-05-21T04:55:38.053Z] CITCMD: ci-setup-anvil-bm-vm [2026-05-21T04:55:38.053Z] CITCMDARGS: rhel-9 ci ci all [2026-05-21T04:55:38.400Z] Genering ssh keys and ssh config [2026-05-21T04:55:38.400Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:55:38.400Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:55:38.400Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:55:38.400Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:55:38.748Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:55:38.748Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-05-21T04:58:47.926Z] Destroying VMs (if any) [2026-05-21T04:58:47.926Z] Warning: Permanently added 'an-striker01' (ED25519) to the list of known hosts. [2026-05-21T04:58:47.926Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-05-21T04:58:47.926Z] This might take a few seconds. [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T04:58:47.926Z] ! Cannot disconnect from Red Hat Insights [2026-05-21T04:58:47.926Z] ! Cannot disconnect from Red Hat Subscription Management [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] Manage your connected systems: https://red.ht/connector [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] The following errors were encountered during disconnect: [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] TYPE STEP ERROR [2026-05-21T04:58:47.926Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-05-21T04:58:47.926Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-05-21T04:58:47.926Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] 1. Register with subscription-manager [2026-05-21T04:58:47.926Z] # subscription-manager register [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] 2. Register with insights-client [2026-05-21T04:58:47.926Z] # insights-client --register [2026-05-21T04:58:47.926Z] This system is currently not registered. [2026-05-21T04:58:47.926Z] Domain 'an-striker01' destroyed [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] Domain 'an-striker01' has been undefined [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] Warning: Permanently added 'an-striker02' (ED25519) to the list of known hosts. [2026-05-21T04:58:47.926Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-05-21T04:58:47.926Z] This might take a few seconds. [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T04:58:47.926Z] ! Cannot disconnect from Red Hat Insights [2026-05-21T04:58:47.926Z] ! Cannot disconnect from Red Hat Subscription Management [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] Manage your connected systems: https://red.ht/connector [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] The following errors were encountered during disconnect: [2026-05-21T04:58:47.926Z] [2026-05-21T04:58:47.926Z] TYPE STEP ERROR [2026-05-21T04:58:47.926Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-05-21T04:58:47.926Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-05-21T04:58:49.129Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T04:58:49.130Z] [2026-05-21T04:58:49.130Z] 1. Register with subscription-manager [2026-05-21T04:58:49.130Z] # subscription-manager register [2026-05-21T04:58:49.130Z] [2026-05-21T04:58:49.130Z] 2. Register with insights-client [2026-05-21T04:58:49.130Z] # insights-client --register [2026-05-21T04:58:50.060Z] This system is currently not registered. [2026-05-21T04:58:52.407Z] Domain 'an-striker02' destroyed [2026-05-21T04:58:52.407Z] [2026-05-21T04:58:52.407Z] Domain 'an-striker02' has been undefined [2026-05-21T04:58:52.407Z] [2026-05-21T04:58:52.407Z] Warning: Permanently added 'an-a01n01' (ED25519) to the list of known hosts. [2026-05-21T04:58:52.754Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-05-21T04:58:52.754Z] This might take a few seconds. [2026-05-21T04:58:52.754Z] [2026-05-21T04:58:52.754Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T04:58:53.957Z] ! Cannot disconnect from Red Hat Insights [2026-05-21T04:58:53.957Z] ! Cannot disconnect from Red Hat Subscription Management [2026-05-21T04:58:53.957Z] [2026-05-21T04:58:53.957Z] Manage your connected systems: https://red.ht/connector [2026-05-21T04:58:53.957Z] [2026-05-21T04:58:53.957Z] The following errors were encountered during disconnect: [2026-05-21T04:58:53.957Z] [2026-05-21T04:58:53.957Z] TYPE STEP ERROR [2026-05-21T04:58:53.957Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-05-21T04:58:53.957Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-05-21T04:58:55.159Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T04:58:55.159Z] [2026-05-21T04:58:55.159Z] 1. Register with subscription-manager [2026-05-21T04:58:55.159Z] # subscription-manager register [2026-05-21T04:58:55.159Z] [2026-05-21T04:58:55.159Z] 2. Register with insights-client [2026-05-21T04:58:55.159Z] # insights-client --register [2026-05-21T04:58:55.904Z] This system is currently not registered. [2026-05-21T04:58:58.969Z] Domain 'an-a01n01' destroyed [2026-05-21T04:58:58.969Z] [2026-05-21T04:58:58.969Z] Domain 'an-a01n01' has been undefined [2026-05-21T04:58:58.969Z] [2026-05-21T04:58:58.969Z] Warning: Permanently added 'an-a01n02' (ED25519) to the list of known hosts. [2026-05-21T04:58:58.969Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-05-21T04:58:58.969Z] This might take a few seconds. [2026-05-21T04:58:58.969Z] [2026-05-21T04:58:58.969Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T04:59:00.172Z] ! Cannot disconnect from Red Hat Insights [2026-05-21T04:59:00.172Z] ! Cannot disconnect from Red Hat Subscription Management [2026-05-21T04:59:00.172Z] [2026-05-21T04:59:00.172Z] Manage your connected systems: https://red.ht/connector [2026-05-21T04:59:00.172Z] [2026-05-21T04:59:00.172Z] The following errors were encountered during disconnect: [2026-05-21T04:59:00.172Z] [2026-05-21T04:59:00.172Z] TYPE STEP ERROR [2026-05-21T04:59:00.172Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-05-21T04:59:00.172Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-05-21T04:59:01.374Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T04:59:01.374Z] [2026-05-21T04:59:01.374Z] 1. Register with subscription-manager [2026-05-21T04:59:01.374Z] # subscription-manager register [2026-05-21T04:59:01.374Z] [2026-05-21T04:59:01.374Z] 2. Register with insights-client [2026-05-21T04:59:01.374Z] # insights-client --register [2026-05-21T04:59:02.119Z] This system is currently not registered. [2026-05-21T04:59:05.348Z] Domain 'an-a01n02' destroyed [2026-05-21T04:59:05.348Z] [2026-05-21T04:59:05.348Z] Domain 'an-a01n02' has been undefined [2026-05-21T04:59:05.348Z] [2026-05-21T04:59:05.348Z] Warning: Permanently added 'an-a01dr01' (ED25519) to the list of known hosts. [2026-05-21T04:59:05.348Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-05-21T04:59:05.348Z] This might take a few seconds. [2026-05-21T04:59:05.348Z] [2026-05-21T04:59:05.348Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T04:59:06.550Z] ! Cannot disconnect from Red Hat Insights [2026-05-21T04:59:06.550Z] ! Cannot disconnect from Red Hat Subscription Management [2026-05-21T04:59:06.550Z] [2026-05-21T04:59:06.550Z] Manage your connected systems: https://red.ht/connector [2026-05-21T04:59:06.550Z] [2026-05-21T04:59:06.550Z] The following errors were encountered during disconnect: [2026-05-21T04:59:06.550Z] [2026-05-21T04:59:06.550Z] TYPE STEP ERROR [2026-05-21T04:59:06.550Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-05-21T04:59:06.550Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-05-21T04:59:07.295Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T04:59:07.295Z] [2026-05-21T04:59:07.295Z] 1. Register with subscription-manager [2026-05-21T04:59:07.295Z] # subscription-manager register [2026-05-21T04:59:07.295Z] [2026-05-21T04:59:07.295Z] 2. Register with insights-client [2026-05-21T04:59:07.295Z] # insights-client --register [2026-05-21T04:59:08.496Z] This system is currently not registered. [2026-05-21T04:59:10.840Z] Domain 'an-a01dr01' destroyed [2026-05-21T04:59:10.840Z] [2026-05-21T04:59:10.840Z] Domain 'an-a01dr01' has been undefined [2026-05-21T04:59:10.840Z] [2026-05-21T04:59:10.840Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-21T04:59:11.187Z] Genering ssh keys and ssh config [2026-05-21T04:59:11.187Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:59:11.187Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:59:11.534Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:59:11.534Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:59:11.534Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T04:59:11.881Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-21T04:59:11.881Z] receiving incremental file list [2026-05-21T04:59:11.881Z] [2026-05-21T04:59:11.881Z] sent 27 bytes received 442 bytes 938.00 bytes/sec [2026-05-21T04:59:11.881Z] total size is 16,565,796,937 speedup is 35,321,528.65 [2026-05-21T04:59:11.881Z] Generating daily cache [2026-05-21T04:59:11.881Z] Updating daily cache for rhel-9 2026-05-21 [2026-05-21T04:59:11.881Z] Destroying VMs (if any) [2026-05-21T04:59:13.081Z] Formatting 'rhel-9-latest.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/images/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T04:59:13.081Z] wrote an-cache-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T04:59:14.812Z] [2026-05-21T04:59:14.812Z] Starting install... [2026-05-21T04:59:14.812Z] [2026-05-21T04:59:14.812Z] Creating domain... | 00:00 [2026-05-21T04:59:14.812Z] Domain creation completed. [2026-05-21T04:59:14.812Z] Waiting for nodes and cloud-init to complete full init [2026-05-21T04:59:17.875Z] cloud-init status node an-cache: [2026-05-21T04:59:17.875Z] Thu May 21 00:59:17 EDT 2026 actual: 0 expected: 1 [2026-05-21T04:59:31.397Z] cloud-init status node an-cache: [2026-05-21T04:59:31.397Z] Thu May 21 00:59:29 EDT 2026 actual: 0 expected: 1 [2026-05-21T04:59:40.643Z] cloud-init status node an-cache: running [2026-05-21T04:59:40.643Z] Thu May 21 00:59:40 EDT 2026 actual: 0 expected: 1 [2026-05-21T04:59:51.687Z] cloud-init status node an-cache: running [2026-05-21T04:59:51.687Z] Thu May 21 00:59:51 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:02.894Z] cloud-init status node an-cache: running [2026-05-21T05:00:02.894Z] Thu May 21 01:00:02 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:13.939Z] cloud-init status node an-cache: running [2026-05-21T05:00:13.939Z] Thu May 21 01:00:12 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:24.981Z] cloud-init status node an-cache: running [2026-05-21T05:00:24.981Z] Thu May 21 01:00:23 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:36.058Z] cloud-init status node an-cache: running [2026-05-21T05:00:36.058Z] Thu May 21 01:00:34 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:45.181Z] cloud-init status node an-cache: running [2026-05-21T05:00:45.181Z] Thu May 21 01:00:44 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:00:56.226Z] cloud-init status node an-cache: running [2026-05-21T05:00:56.226Z] Thu May 21 01:00:55 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:01:07.495Z] cloud-init status node an-cache: running [2026-05-21T05:01:07.495Z] Thu May 21 01:01:06 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:01:18.903Z] cloud-init status node an-cache: running [2026-05-21T05:01:18.903Z] Thu May 21 01:01:17 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:01:28.387Z] cloud-init status node an-cache: running [2026-05-21T05:01:28.387Z] Thu May 21 01:01:28 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:01:39.132Z] cloud-init status node an-cache: running [2026-05-21T05:01:39.133Z] Thu May 21 01:01:39 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:01:50.362Z] cloud-init status node an-cache: running [2026-05-21T05:01:50.362Z] Thu May 21 01:01:49 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:01.405Z] cloud-init status node an-cache: running [2026-05-21T05:02:01.405Z] Thu May 21 01:02:00 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:13.402Z] cloud-init status node an-cache: running [2026-05-21T05:02:13.402Z] Thu May 21 01:02:11 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:22.491Z] cloud-init status node an-cache: running [2026-05-21T05:02:22.491Z] Thu May 21 01:02:22 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:33.528Z] cloud-init status node an-cache: running [2026-05-21T05:02:33.528Z] Thu May 21 01:02:32 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:44.567Z] cloud-init status node an-cache: running [2026-05-21T05:02:44.567Z] Thu May 21 01:02:43 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:02:55.605Z] cloud-init status node an-cache: running [2026-05-21T05:02:55.605Z] Thu May 21 01:02:54 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:03:06.647Z] cloud-init status node an-cache: running [2026-05-21T05:03:06.647Z] Thu May 21 01:03:05 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:03:16.115Z] cloud-init status node an-cache: running [2026-05-21T05:03:16.115Z] Thu May 21 01:03:15 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:03:27.152Z] cloud-init status node an-cache: running [2026-05-21T05:03:27.152Z] Thu May 21 01:03:26 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:03:38.187Z] cloud-init status node an-cache: running [2026-05-21T05:03:38.187Z] Thu May 21 01:03:37 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:03:49.602Z] cloud-init status node an-cache: running [2026-05-21T05:03:49.602Z] Thu May 21 01:03:48 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:00.254Z] cloud-init status node an-cache: running [2026-05-21T05:04:00.254Z] Thu May 21 01:03:59 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:09.383Z] cloud-init status node an-cache: [2026-05-21T05:04:09.383Z] Thu May 21 01:04:09 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:20.295Z] cloud-init status node an-cache: [2026-05-21T05:04:20.295Z] Thu May 21 01:04:19 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:30.769Z] cloud-init status node an-cache: [2026-05-21T05:04:30.769Z] Thu May 21 01:04:29 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:39.871Z] cloud-init status node an-cache: [2026-05-21T05:04:39.871Z] Thu May 21 01:04:39 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:04:49.919Z] cloud-init status node an-cache: [2026-05-21T05:04:49.919Z] Thu May 21 01:04:49 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:00.106Z] cloud-init status node an-cache: [2026-05-21T05:05:00.106Z] Thu May 21 01:04:59 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:09.614Z] cloud-init status node an-cache: [2026-05-21T05:05:09.614Z] Thu May 21 01:05:09 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:19.994Z] cloud-init status node an-cache: [2026-05-21T05:05:19.994Z] Thu May 21 01:05:19 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:31.702Z] cloud-init status node an-cache: running [2026-05-21T05:05:31.702Z] Thu May 21 01:05:30 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:41.183Z] cloud-init status node an-cache: running [2026-05-21T05:05:41.183Z] Thu May 21 01:05:40 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:05:52.270Z] cloud-init status node an-cache: running [2026-05-21T05:05:52.270Z] Thu May 21 01:05:51 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:03.311Z] cloud-init status node an-cache: running [2026-05-21T05:06:03.311Z] Thu May 21 01:06:02 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:14.350Z] cloud-init status node an-cache: running [2026-05-21T05:06:14.350Z] Thu May 21 01:06:12 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:25.393Z] cloud-init status node an-cache: running [2026-05-21T05:06:25.393Z] Thu May 21 01:06:23 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:34.999Z] cloud-init status node an-cache: running [2026-05-21T05:06:34.999Z] Thu May 21 01:06:34 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:46.128Z] cloud-init status node an-cache: running [2026-05-21T05:06:46.128Z] Thu May 21 01:06:45 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:06:57.200Z] cloud-init status node an-cache: running [2026-05-21T05:06:57.200Z] Thu May 21 01:06:56 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:07:07.437Z] cloud-init status node an-cache: running [2026-05-21T05:07:07.437Z] Thu May 21 01:07:07 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:07:18.514Z] cloud-init status node an-cache: running [2026-05-21T05:07:18.514Z] Thu May 21 01:07:17 EDT 2026 actual: 0 expected: 1 [2026-05-21T05:07:29.686Z] cloud-init status node an-cache: done [2026-05-21T05:07:29.686Z] Thu May 21 01:07:28 EDT 2026 actual: 1 expected: 1 [2026-05-21T05:08:54.866Z] Disconnecting an-cache.ci.alteeve.com from Red Hat. [2026-05-21T05:08:54.866Z] This might take a few seconds. [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T05:08:54.866Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T05:08:54.866Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] Manage your connected systems: https://red.ht/connector [2026-05-21T05:08:54.866Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] 1. Register with subscription-manager [2026-05-21T05:08:54.866Z] # subscription-manager register [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] 2. Register with insights-client [2026-05-21T05:08:54.866Z] # insights-client --register [2026-05-21T05:08:54.866Z] This system is currently not registered. [2026-05-21T05:08:54.866Z] Domain 'an-cache' is being shutdown [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] Domain 'an-cache' has been undefined [2026-05-21T05:08:54.866Z] [2026-05-21T05:08:54.866Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T05:08:54.866Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T05:08:54.866Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T05:08:54.866Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T05:08:54.866Z] # github.com:22 SSH-2.0-8ad108e [2026-05-21T05:08:54.866Z] Done updating daily cache [2026-05-21T05:08:54.866Z] Creating nodes disks [2026-05-21T05:09:02.324Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-21T05:09:02.324Z] Generating an-test-deployX kickstart files/isos [2026-05-21T05:09:02.324Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-05-21T05:09:02.324Z] Media current: stdio file, overwriteable [2026-05-21T05:09:02.324Z] Media status : is blank [2026-05-21T05:09:02.324Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1728g free [2026-05-21T05:09:02.324Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] ISO image produced: 184 sectors [2026-05-21T05:09:02.324Z] Written to medium : 184 sectors at LBA 0 [2026-05-21T05:09:02.324Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-05-21T05:09:02.324Z] Media current: stdio file, overwriteable [2026-05-21T05:09:02.324Z] Media status : is blank [2026-05-21T05:09:02.324Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1728g free [2026-05-21T05:09:02.324Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] ISO image produced: 184 sectors [2026-05-21T05:09:02.324Z] Written to medium : 184 sectors at LBA 0 [2026-05-21T05:09:02.324Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-05-21T05:09:02.324Z] Media current: stdio file, overwriteable [2026-05-21T05:09:02.324Z] Media status : is blank [2026-05-21T05:09:02.324Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1728g free [2026-05-21T05:09:02.324Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] ISO image produced: 184 sectors [2026-05-21T05:09:02.324Z] Written to medium : 184 sectors at LBA 0 [2026-05-21T05:09:02.324Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-05-21T05:09:02.324Z] Media current: stdio file, overwriteable [2026-05-21T05:09:02.324Z] Media status : is blank [2026-05-21T05:09:02.324Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1728g free [2026-05-21T05:09:02.324Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] ISO image produced: 184 sectors [2026-05-21T05:09:02.324Z] Written to medium : 184 sectors at LBA 0 [2026-05-21T05:09:02.324Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-05-21T05:09:02.324Z] Media current: stdio file, overwriteable [2026-05-21T05:09:02.324Z] Media status : is blank [2026-05-21T05:09:02.324Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1728g free [2026-05-21T05:09:02.324Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-21T05:09:02.324Z] ISO image produced: 184 sectors [2026-05-21T05:09:02.324Z] Written to medium : 184 sectors at LBA 0 [2026-05-21T05:09:02.324Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-05-21T05:09:02.324Z] [2026-05-21T05:09:02.324Z] Generating cloud-init configs [2026-05-21T05:09:02.324Z] Generating cloud-init metadata and seed images [2026-05-21T05:09:02.324Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T05:09:02.324Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T05:09:02.673Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T05:09:02.673Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T05:09:02.673Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-21T05:09:02.673Z] Creating VMs [2026-05-21T05:09:04.406Z] [2026-05-21T05:09:04.406Z] Starting install... [2026-05-21T05:09:04.406Z] [2026-05-21T05:09:04.406Z] Creating domain... | 00:00 [2026-05-21T05:09:04.406Z] Domain creation completed. [2026-05-21T05:09:06.142Z] [2026-05-21T05:09:06.142Z] Starting install... [2026-05-21T05:09:06.142Z] [2026-05-21T05:09:06.142Z] Creating domain... | 00:00 [2026-05-21T05:09:06.142Z] Domain creation completed. [2026-05-21T05:09:08.006Z] [2026-05-21T05:09:08.006Z] Starting install... [2026-05-21T05:09:08.006Z] [2026-05-21T05:09:08.006Z] Creating domain... | 00:00 [2026-05-21T05:09:08.006Z] Domain creation completed. [2026-05-21T05:09:09.208Z] [2026-05-21T05:09:09.208Z] Starting install... [2026-05-21T05:09:09.208Z] [2026-05-21T05:09:09.208Z] Creating domain... | 00:00 [2026-05-21T05:09:09.208Z] Domain creation completed. [2026-05-21T05:09:11.556Z] [2026-05-21T05:09:11.556Z] Starting install... [2026-05-21T05:09:11.556Z] [2026-05-21T05:09:11.556Z] Creating domain... | 00:00 [2026-05-21T05:09:11.556Z] Domain creation completed. [2026-05-21T05:09:11.556Z] Waiting for nodes and cloud-init to complete full init [2026-05-21T05:09:14.699Z] cloud-init status node an-striker01: [2026-05-21T05:09:17.911Z] cloud-init status node an-striker02: [2026-05-21T05:09:20.359Z] cloud-init status node an-a01n01: [2026-05-21T05:09:23.446Z] cloud-init status node an-a01n02: [2026-05-21T05:09:26.496Z] cloud-init status node an-a01dr01: [2026-05-21T05:09:26.496Z] Thu May 21 01:09:26 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:09:39.864Z] cloud-init status node an-striker01: running [2026-05-21T05:09:39.864Z] cloud-init status node an-striker02: running [2026-05-21T05:09:40.213Z] cloud-init status node an-a01n01: running [2026-05-21T05:09:41.417Z] cloud-init status node an-a01n02: running [2026-05-21T05:09:42.623Z] cloud-init status node an-a01dr01: running [2026-05-21T05:09:42.623Z] Thu May 21 01:09:42 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:09:53.673Z] cloud-init status node an-striker01: running [2026-05-21T05:09:54.421Z] cloud-init status node an-striker02: running [2026-05-21T05:09:55.468Z] cloud-init status node an-a01n01: running [2026-05-21T05:09:56.213Z] cloud-init status node an-a01n02: running [2026-05-21T05:09:56.958Z] cloud-init status node an-a01dr01: running [2026-05-21T05:09:56.958Z] Thu May 21 01:09:56 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:10:07.999Z] cloud-init status node an-striker01: running [2026-05-21T05:10:08.744Z] cloud-init status node an-striker02: running [2026-05-21T05:10:09.948Z] cloud-init status node an-a01n01: running [2026-05-21T05:10:10.694Z] cloud-init status node an-a01n02: running [2026-05-21T05:10:11.438Z] cloud-init status node an-a01dr01: running [2026-05-21T05:10:11.438Z] Thu May 21 01:10:11 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:10:22.810Z] cloud-init status node an-striker01: running [2026-05-21T05:10:23.158Z] cloud-init status node an-striker02: running [2026-05-21T05:10:24.015Z] cloud-init status node an-a01n01: running [2026-05-21T05:10:25.028Z] cloud-init status node an-a01n02: running [2026-05-21T05:10:25.869Z] cloud-init status node an-a01dr01: running [2026-05-21T05:10:25.869Z] Thu May 21 01:10:25 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:10:37.130Z] cloud-init status node an-striker01: running [2026-05-21T05:10:38.641Z] cloud-init status node an-striker02: running [2026-05-21T05:10:45.036Z] cloud-init status node an-a01n01: running [2026-05-21T05:10:46.236Z] cloud-init status node an-a01n02: running [2026-05-21T05:10:46.591Z] cloud-init status node an-a01dr01: running [2026-05-21T05:10:46.591Z] Thu May 21 01:10:46 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:10:58.232Z] cloud-init status node an-striker01: running [2026-05-21T05:10:58.744Z] cloud-init status node an-striker02: running [2026-05-21T05:11:00.130Z] cloud-init status node an-a01n01: running [2026-05-21T05:11:00.918Z] cloud-init status node an-a01n02: running [2026-05-21T05:11:01.730Z] cloud-init status node an-a01dr01: running [2026-05-21T05:11:01.730Z] Thu May 21 01:11:01 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:11:13.112Z] cloud-init status node an-striker01: running [2026-05-21T05:11:14.180Z] cloud-init status node an-striker02: running [2026-05-21T05:11:20.670Z] cloud-init status node an-a01n01: running [2026-05-21T05:11:20.670Z] cloud-init status node an-a01n02: running [2026-05-21T05:11:21.678Z] cloud-init status node an-a01dr01: running [2026-05-21T05:11:21.678Z] Thu May 21 01:11:21 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:11:32.742Z] cloud-init status node an-striker01: running [2026-05-21T05:11:33.486Z] cloud-init status node an-striker02: running [2026-05-21T05:11:34.230Z] cloud-init status node an-a01n01: running [2026-05-21T05:11:35.434Z] cloud-init status node an-a01n02: running [2026-05-21T05:11:36.178Z] cloud-init status node an-a01dr01: running [2026-05-21T05:11:36.178Z] Thu May 21 01:11:35 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:11:47.516Z] cloud-init status node an-striker01: running [2026-05-21T05:11:47.864Z] cloud-init status node an-striker02: running [2026-05-21T05:11:48.608Z] cloud-init status node an-a01n01: running [2026-05-21T05:11:49.811Z] cloud-init status node an-a01n02: running [2026-05-21T05:11:50.557Z] cloud-init status node an-a01dr01: running [2026-05-21T05:11:50.557Z] Thu May 21 01:11:50 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:12:01.598Z] cloud-init status node an-striker01: running [2026-05-21T05:12:02.344Z] cloud-init status node an-striker02: running [2026-05-21T05:12:03.089Z] cloud-init status node an-a01n01: running [2026-05-21T05:12:04.293Z] cloud-init status node an-a01n02: running [2026-05-21T05:12:05.040Z] cloud-init status node an-a01dr01: running [2026-05-21T05:12:05.040Z] Thu May 21 01:12:04 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:12:16.337Z] cloud-init status node an-striker01: running [2026-05-21T05:12:17.085Z] cloud-init status node an-striker02: running [2026-05-21T05:12:17.833Z] cloud-init status node an-a01n01: running [2026-05-21T05:12:18.578Z] cloud-init status node an-a01n02: running [2026-05-21T05:12:19.782Z] cloud-init status node an-a01dr01: running [2026-05-21T05:12:19.782Z] Thu May 21 01:12:19 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:12:30.826Z] cloud-init status node an-striker01: running [2026-05-21T05:12:31.571Z] cloud-init status node an-striker02: running [2026-05-21T05:12:32.319Z] cloud-init status node an-a01n01: running [2026-05-21T05:12:33.229Z] cloud-init status node an-a01n02: running [2026-05-21T05:12:33.975Z] cloud-init status node an-a01dr01: running [2026-05-21T05:12:34.323Z] Thu May 21 01:12:33 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:12:45.371Z] cloud-init status node an-striker01: running [2026-05-21T05:12:46.117Z] cloud-init status node an-striker02: running [2026-05-21T05:12:46.863Z] cloud-init status node an-a01n01: running [2026-05-21T05:12:48.065Z] cloud-init status node an-a01n02: running [2026-05-21T05:12:48.810Z] cloud-init status node an-a01dr01: running [2026-05-21T05:12:48.810Z] Thu May 21 01:12:48 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:12:59.857Z] cloud-init status node an-striker01: running [2026-05-21T05:13:00.603Z] cloud-init status node an-striker02: running [2026-05-21T05:13:01.807Z] cloud-init status node an-a01n01: running [2026-05-21T05:13:02.552Z] cloud-init status node an-a01n02: running [2026-05-21T05:13:03.755Z] cloud-init status node an-a01dr01: running [2026-05-21T05:13:03.755Z] Thu May 21 01:13:03 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:13:14.798Z] cloud-init status node an-striker01: running [2026-05-21T05:13:15.542Z] cloud-init status node an-striker02: running [2026-05-21T05:13:16.287Z] cloud-init status node an-a01n01: running [2026-05-21T05:13:17.490Z] cloud-init status node an-a01n02: running [2026-05-21T05:13:18.235Z] cloud-init status node an-a01dr01: running [2026-05-21T05:13:18.235Z] Thu May 21 01:13:17 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:13:29.423Z] cloud-init status node an-striker01: running [2026-05-21T05:13:30.168Z] cloud-init status node an-striker02: running [2026-05-21T05:13:30.913Z] cloud-init status node an-a01n01: running [2026-05-21T05:13:32.115Z] cloud-init status node an-a01n02: running [2026-05-21T05:13:32.859Z] cloud-init status node an-a01dr01: running [2026-05-21T05:13:32.859Z] Thu May 21 01:13:32 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:13:43.900Z] cloud-init status node an-striker01: running [2026-05-21T05:13:44.644Z] cloud-init status node an-striker02: running [2026-05-21T05:13:46.135Z] cloud-init status node an-a01n01: running [2026-05-21T05:13:46.483Z] cloud-init status node an-a01n02: running [2026-05-21T05:13:47.687Z] cloud-init status node an-a01dr01: running [2026-05-21T05:13:47.687Z] Thu May 21 01:13:47 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:13:58.729Z] cloud-init status node an-striker01: running [2026-05-21T05:13:59.475Z] cloud-init status node an-striker02: running [2026-05-21T05:14:00.220Z] cloud-init status node an-a01n01: running [2026-05-21T05:14:01.422Z] cloud-init status node an-a01n02: running [2026-05-21T05:14:02.167Z] cloud-init status node an-a01dr01: running [2026-05-21T05:14:02.167Z] Thu May 21 01:14:01 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:14:13.211Z] cloud-init status node an-striker01: running [2026-05-21T05:14:13.956Z] cloud-init status node an-striker02: running [2026-05-21T05:14:15.159Z] cloud-init status node an-a01n01: running [2026-05-21T05:14:16.363Z] cloud-init status node an-a01n02: running [2026-05-21T05:14:17.109Z] cloud-init status node an-a01dr01: running [2026-05-21T05:14:17.109Z] Thu May 21 01:14:16 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:14:28.153Z] cloud-init status node an-striker01: running [2026-05-21T05:14:28.899Z] cloud-init status node an-striker02: running [2026-05-21T05:14:30.102Z] cloud-init status node an-a01n01: running [2026-05-21T05:14:30.846Z] cloud-init status node an-a01n02: running [2026-05-21T05:14:31.591Z] cloud-init status node an-a01dr01: running [2026-05-21T05:14:31.591Z] Thu May 21 01:14:31 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:14:42.886Z] cloud-init status node an-striker01: running [2026-05-21T05:14:43.630Z] cloud-init status node an-striker02: running [2026-05-21T05:14:44.375Z] cloud-init status node an-a01n01: running [2026-05-21T05:14:45.576Z] cloud-init status node an-a01n02: running [2026-05-21T05:14:46.321Z] cloud-init status node an-a01dr01: running [2026-05-21T05:14:46.321Z] Thu May 21 01:14:46 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:14:57.360Z] cloud-init status node an-striker01: running [2026-05-21T05:14:58.105Z] cloud-init status node an-striker02: running [2026-05-21T05:14:59.530Z] cloud-init status node an-a01n01: running [2026-05-21T05:15:00.275Z] cloud-init status node an-a01n02: running [2026-05-21T05:15:01.478Z] cloud-init status node an-a01dr01: running [2026-05-21T05:15:01.478Z] Thu May 21 01:15:01 EDT 2026 actual: 0 expected: 5 [2026-05-21T05:15:14.846Z] cloud-init status node an-striker01: running [2026-05-21T05:15:14.846Z] cloud-init status node an-striker02: running [2026-05-21T05:15:15.594Z] cloud-init status node an-a01n01: running [2026-05-21T05:15:16.799Z] cloud-init status node an-a01n02: running [2026-05-21T05:15:18.535Z] cloud-init status node an-a01dr01: done [2026-05-21T05:15:18.535Z] Thu May 21 01:15:18 EDT 2026 actual: 1 expected: 5 [2026-05-21T05:15:29.708Z] cloud-init status node an-striker01: running [2026-05-21T05:15:30.913Z] cloud-init status node an-striker02: running [2026-05-21T05:15:32.117Z] cloud-init status node an-a01n01: done [2026-05-21T05:15:33.322Z] cloud-init status node an-a01n02: done [2026-05-21T05:15:34.526Z] cloud-init status node an-a01dr01: done [2026-05-21T05:15:34.526Z] Thu May 21 01:15:34 EDT 2026 actual: 3 expected: 5 [2026-05-21T05:15:45.602Z] cloud-init status node an-striker01: running [2026-05-21T05:15:47.333Z] cloud-init status node an-striker02: running [2026-05-21T05:15:48.680Z] cloud-init status node an-a01n01: done [2026-05-21T05:15:49.428Z] cloud-init status node an-a01n02: done [2026-05-21T05:15:50.175Z] cloud-init status node an-a01dr01: done [2026-05-21T05:15:50.175Z] Thu May 21 01:15:50 EDT 2026 actual: 3 expected: 5 [2026-05-21T05:16:01.216Z] cloud-init status node an-striker01: running [2026-05-21T05:16:02.418Z] cloud-init status node an-striker02: running [2026-05-21T05:16:03.621Z] cloud-init status node an-a01n01: done [2026-05-21T05:16:04.366Z] cloud-init status node an-a01n02: done [2026-05-21T05:16:05.111Z] cloud-init status node an-a01dr01: done [2026-05-21T05:16:05.111Z] Thu May 21 01:16:04 EDT 2026 actual: 3 expected: 5 [2026-05-21T05:16:16.296Z] cloud-init status node an-striker01: running [2026-05-21T05:16:17.498Z] cloud-init status node an-striker02: running [2026-05-21T05:16:18.243Z] cloud-init status node an-a01n01: done [2026-05-21T05:16:18.988Z] cloud-init status node an-a01n02: done [2026-05-21T05:16:20.191Z] cloud-init status node an-a01dr01: done [2026-05-21T05:16:20.191Z] Thu May 21 01:16:19 EDT 2026 actual: 3 expected: 5 [2026-05-21T05:16:31.234Z] cloud-init status node an-striker01: running [2026-05-21T05:16:31.979Z] cloud-init status node an-striker02: running [2026-05-21T05:16:33.183Z] cloud-init status node an-a01n01: done [2026-05-21T05:16:33.929Z] cloud-init status node an-a01n02: done [2026-05-21T05:16:35.132Z] cloud-init status node an-a01dr01: done [2026-05-21T05:16:35.132Z] Thu May 21 01:16:34 EDT 2026 actual: 3 expected: 5 [2026-05-21T05:16:46.318Z] cloud-init status node an-striker01: done [2026-05-21T05:16:46.918Z] cloud-init status node an-striker02: running [2026-05-21T05:16:47.689Z] cloud-init status node an-a01n01: done [2026-05-21T05:16:48.892Z] cloud-init status node an-a01n02: done [2026-05-21T05:16:49.641Z] cloud-init status node an-a01dr01: done [2026-05-21T05:16:49.641Z] Thu May 21 01:16:49 EDT 2026 actual: 4 expected: 5 [2026-05-21T05:17:01.291Z] cloud-init status node an-striker01: done [2026-05-21T05:17:03.522Z] cloud-init status node an-striker02: running [2026-05-21T05:17:03.891Z] cloud-init status node an-a01n01: done [2026-05-21T05:17:05.216Z] cloud-init status node an-a01n02: done [2026-05-21T05:17:05.964Z] cloud-init status node an-a01dr01: done [2026-05-21T05:17:05.964Z] Thu May 21 01:17:05 EDT 2026 actual: 4 expected: 5 [2026-05-21T05:17:17.361Z] cloud-init status node an-striker01: done [2026-05-21T05:17:18.151Z] cloud-init status node an-striker02: done [2026-05-21T05:17:18.902Z] cloud-init status node an-a01n01: done [2026-05-21T05:17:19.658Z] cloud-init status node an-a01n02: done [2026-05-21T05:17:20.867Z] cloud-init status node an-a01dr01: done [2026-05-21T05:17:20.867Z] Thu May 21 01:17:20 EDT 2026 actual: 5 expected: 5 [2026-05-21T05:17:20.867Z] Building simengine model [2026-05-21T05:17:34.366Z] Generating anvil striker auto config [2026-05-21T05:17:36.099Z] Cloning into 'anvil-test-suite'... [2026-05-21T05:17:36.845Z] generated ansible config: [2026-05-21T05:17:36.845Z] ========================= [2026-05-21T05:17:36.845Z] [defaults] [2026-05-21T05:17:36.845Z] inventory = ansible-inventory [2026-05-21T05:17:36.845Z] forks = 500 [2026-05-21T05:17:36.845Z] any_errors_fatal = true [2026-05-21T05:17:36.845Z] display_args_to_stdout = true [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:36.845Z] [ssh_connection] [2026-05-21T05:17:36.845Z] retries = 20 [2026-05-21T05:17:36.845Z] ========================= [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:36.845Z] generated ansible inventory: [2026-05-21T05:17:36.845Z] ============================ [2026-05-21T05:17:36.845Z] [strikers] [2026-05-21T05:17:36.845Z] an-striker01 [2026-05-21T05:17:36.845Z] an-striker02 [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:36.845Z] [nodes] [2026-05-21T05:17:36.845Z] an-a01n01 [2026-05-21T05:17:36.845Z] an-a01n02 [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:36.845Z] [drnodes] [2026-05-21T05:17:36.845Z] an-a01dr01 [2026-05-21T05:17:36.845Z] ============================ [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:36.845Z] Checking connectivity to nodes [2026-05-21T05:17:36.845Z] [2026-05-21T05:17:38.578Z] an-a01n01 | SUCCESS => { [2026-05-21T05:17:38.578Z] "ansible_facts": { [2026-05-21T05:17:38.578Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-21T05:17:38.578Z] }, [2026-05-21T05:17:38.578Z] "changed": false, [2026-05-21T05:17:38.578Z] "ping": "pong" [2026-05-21T05:17:38.578Z] } [2026-05-21T05:17:38.578Z] an-a01dr01 | SUCCESS => { [2026-05-21T05:17:38.578Z] "ansible_facts": { [2026-05-21T05:17:38.578Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-21T05:17:38.578Z] }, [2026-05-21T05:17:38.578Z] "changed": false, [2026-05-21T05:17:38.578Z] "ping": "pong" [2026-05-21T05:17:38.578Z] } [2026-05-21T05:17:38.578Z] an-a01n02 | SUCCESS => { [2026-05-21T05:17:38.578Z] "ansible_facts": { [2026-05-21T05:17:38.578Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-21T05:17:38.578Z] }, [2026-05-21T05:17:38.578Z] "changed": false, [2026-05-21T05:17:38.578Z] "ping": "pong" [2026-05-21T05:17:38.578Z] } [2026-05-21T05:17:38.578Z] an-striker02 | SUCCESS => { [2026-05-21T05:17:38.578Z] "ansible_facts": { [2026-05-21T05:17:38.578Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-21T05:17:38.578Z] }, [2026-05-21T05:17:38.578Z] "changed": false, [2026-05-21T05:17:38.578Z] "ping": "pong" [2026-05-21T05:17:38.578Z] } [2026-05-21T05:17:38.578Z] an-striker01 | SUCCESS => { [2026-05-21T05:17:38.578Z] "ansible_facts": { [2026-05-21T05:17:38.578Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-21T05:17:38.578Z] }, [2026-05-21T05:17:38.578Z] "changed": false, [2026-05-21T05:17:38.578Z] "ping": "pong" [2026-05-21T05:17:38.578Z] } [2026-05-21T05:17:38.578Z] [2026-05-21T05:17:39.781Z] [2026-05-21T05:17:39.781Z] PLAY [Setup all nodes] ********************************************************* [2026-05-21T05:17:39.781Z] [2026-05-21T05:17:39.781Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:17:42.130Z] ok: [an-a01n02] [2026-05-21T05:17:42.130Z] ok: [an-a01n01] [2026-05-21T05:17:42.130Z] ok: [an-a01dr01] [2026-05-21T05:17:42.130Z] ok: [an-striker01] [2026-05-21T05:17:42.130Z] ok: [an-striker02] [2026-05-21T05:17:42.130Z] [2026-05-21T05:17:42.130Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-05-21T05:17:42.875Z] changed: [an-a01n01] [2026-05-21T05:17:42.875Z] changed: [an-a01n02] [2026-05-21T05:17:42.875Z] changed: [an-a01dr01] [2026-05-21T05:17:42.875Z] changed: [an-striker02] [2026-05-21T05:17:42.875Z] changed: [an-striker01] [2026-05-21T05:17:42.875Z] [2026-05-21T05:17:42.875Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-05-21T05:17:43.620Z] ok: [an-a01n02] [2026-05-21T05:17:43.620Z] ok: [an-a01n01] [2026-05-21T05:17:43.620Z] ok: [an-a01dr01] [2026-05-21T05:17:43.620Z] ok: [an-striker02] [2026-05-21T05:17:43.620Z] ok: [an-striker01] [2026-05-21T05:17:43.620Z] [2026-05-21T05:17:43.620Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-05-21T05:17:44.823Z] changed: [an-a01n02] [2026-05-21T05:17:44.823Z] changed: [an-a01n01] [2026-05-21T05:17:44.823Z] changed: [an-a01dr01] [2026-05-21T05:17:44.823Z] changed: [an-striker02] [2026-05-21T05:17:44.823Z] changed: [an-striker01] [2026-05-21T05:17:44.823Z] [2026-05-21T05:17:44.823Z] PLAY [setup strikers] ********************************************************** [2026-05-21T05:17:44.823Z] [2026-05-21T05:17:44.823Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:17:46.554Z] ok: [an-striker02] [2026-05-21T05:17:46.554Z] ok: [an-striker01] [2026-05-21T05:17:46.554Z] [2026-05-21T05:17:46.554Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-05-21T05:17:48.407Z] changed: [an-striker02] [2026-05-21T05:17:48.407Z] changed: [an-striker01] [2026-05-21T05:17:48.407Z] [2026-05-21T05:17:48.407Z] PLAY [setup lvm storage] ******************************************************* [2026-05-21T05:17:48.407Z] [2026-05-21T05:17:48.407Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:17:49.610Z] ok: [an-a01dr01] [2026-05-21T05:17:49.610Z] ok: [an-a01n01] [2026-05-21T05:17:49.610Z] ok: [an-a01n02] [2026-05-21T05:17:49.610Z] [2026-05-21T05:17:49.610Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-05-21T05:17:49.610Z] ] **************** [2026-05-21T05:17:50.355Z] ok: [an-a01n01] [2026-05-21T05:17:50.355Z] ok: [an-a01dr01] [2026-05-21T05:17:50.355Z] ok: [an-a01n02] [2026-05-21T05:17:50.355Z] [2026-05-21T05:17:50.355Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-05-21T05:17:50.355Z] skipping: [an-a01n01] [2026-05-21T05:17:50.355Z] [2026-05-21T05:17:50.355Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-05-21T05:17:51.557Z] changed: [an-a01dr01] [2026-05-21T05:17:51.557Z] changed: [an-a01n01] [2026-05-21T05:17:51.557Z] changed: [an-a01n02] [2026-05-21T05:17:51.557Z] [2026-05-21T05:17:51.557Z] PLAY [deploy anvil] ************************************************************ [2026-05-21T05:17:51.557Z] [2026-05-21T05:17:51.557Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:17:53.289Z] ok: [an-striker02] [2026-05-21T05:17:53.289Z] ok: [an-striker01] [2026-05-21T05:17:53.289Z] [2026-05-21T05:17:53.289Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-05-21T05:17:54.491Z] ok: [an-striker02] [2026-05-21T05:17:54.491Z] ok: [an-striker01] [2026-05-21T05:17:54.491Z] [2026-05-21T05:17:54.491Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-05-21T05:17:56.223Z] changed: [an-striker02] [2026-05-21T05:17:56.223Z] changed: [an-striker01] [2026-05-21T05:17:56.223Z] [2026-05-21T05:17:56.223Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-05-21T05:17:56.223Z] ] *** [2026-05-21T05:17:58.573Z] changed: [an-striker02] [2026-05-21T05:17:58.574Z] changed: [an-striker01] [2026-05-21T05:17:58.574Z] [2026-05-21T05:17:58.574Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:17:58.574Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:17:58.574Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-05-21T05:17:58.574Z] ] *** [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-05-21T05:23:54.047Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-05-21T05:25:17.715Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-05-21T05:25:17.715Z] ok: [an-striker01 -> localhost] [2026-05-21T05:25:17.715Z] ok: [an-striker02 -> localhost] [2026-05-21T05:25:17.715Z] [2026-05-21T05:25:17.715Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-05-21T05:25:17.715Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-05-21T05:25:17.715Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-05-21T05:25:17.715Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-05-21T05:25:17.715Z] echo "DB has completed sync" [2026-05-21T05:25:17.715Z] exit 0 [2026-05-21T05:25:17.715Z] fi [2026-05-21T05:25:17.715Z] if [ "$dbcon" != 2 ]; then [2026-05-21T05:25:17.715Z] echo "Detected $dbcon db connections (expected 2)" [2026-05-21T05:25:17.715Z] exit 1 [2026-05-21T05:25:17.715Z] fi [2026-05-21T05:25:17.715Z] ] *** [2026-05-21T05:25:51.628Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-05-21T05:25:51.628Z] ok: [an-striker01] [2026-05-21T05:25:51.628Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-05-21T05:25:51.628Z] ok: [an-striker02] [2026-05-21T05:25:51.628Z] [2026-05-21T05:25:51.628Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:25:51.628Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:25:51.628Z] --username=postgres anvil -t \ [2026-05-21T05:25:51.628Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-05-21T05:25:51.628Z] grep altee | \ [2026-05-21T05:25:51.628Z] grep 1$ | \ [2026-05-21T05:25:51.628Z] wc -l [2026-05-21T05:25:51.628Z] ] *** [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-05-21T05:31:32.191Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-05-21T05:35:27.035Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-05-21T05:35:27.035Z] ok: [an-striker02 -> localhost] [2026-05-21T05:35:27.035Z] ok: [an-striker01 -> localhost] [2026-05-21T05:35:27.035Z] [2026-05-21T05:35:27.035Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:35:27.035Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:35:27.035Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-05-21T05:35:27.035Z] grep altee | \ [2026-05-21T05:35:27.035Z] wc -l [2026-05-21T05:35:27.035Z] ] *** [2026-05-21T05:35:27.035Z] ok: [an-striker01 -> localhost] [2026-05-21T05:35:27.035Z] ok: [an-striker02 -> localhost] [2026-05-21T05:35:27.035Z] [2026-05-21T05:35:27.035Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:35:27.035Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:35:27.035Z] --username=postgres anvil -t \ [2026-05-21T05:35:27.035Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-05-21T05:35:27.035Z] grep altee | \ [2026-05-21T05:35:27.035Z] grep 1$ | \ [2026-05-21T05:35:27.035Z] wc -l [2026-05-21T05:35:27.035Z] ] *** [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (346 retries left). [2026-05-21T05:37:51.882Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (346 retries left). [2026-05-21T05:37:51.882Z] ok: [an-striker01 -> localhost] [2026-05-21T05:37:51.882Z] ok: [an-striker02 -> localhost] [2026-05-21T05:37:51.882Z] [2026-05-21T05:37:51.882Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:37:51.882Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:37:51.882Z] --username=postgres anvil -t \ [2026-05-21T05:37:51.882Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-05-21T05:37:51.882Z] grep altee | \ [2026-05-21T05:37:51.882Z] grep 1$ | \ [2026-05-21T05:37:51.882Z] wc -l [2026-05-21T05:37:51.882Z] ] *** [2026-05-21T05:38:11.410Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-05-21T05:38:11.410Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-05-21T05:38:11.410Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-05-21T05:38:11.410Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-05-21T05:38:11.410Z] ok: [an-striker01 -> localhost] [2026-05-21T05:38:11.410Z] ok: [an-striker02 -> localhost] [2026-05-21T05:38:11.410Z] [2026-05-21T05:38:11.410Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-21T05:38:11.410Z] psql --host="{{ inventory_hostname }}" \ [2026-05-21T05:38:11.410Z] --username=postgres anvil -t \ [2026-05-21T05:38:11.410Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-05-21T05:38:11.410Z] grep altee | \ [2026-05-21T05:38:11.410Z] grep 1$ | \ [2026-05-21T05:38:11.410Z] wc -l [2026-05-21T05:38:11.410Z] ] *** [2026-05-21T05:38:20.753Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-05-21T05:38:20.753Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-05-21T05:38:20.753Z] ok: [an-striker01 -> localhost] [2026-05-21T05:38:20.753Z] ok: [an-striker02 -> localhost] [2026-05-21T05:38:20.753Z] [2026-05-21T05:38:20.753Z] PLAY [check network status after deployment] *********************************** [2026-05-21T05:38:20.753Z] [2026-05-21T05:38:20.753Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:38:22.526Z] ok: [an-a01dr01] [2026-05-21T05:38:22.526Z] ok: [an-striker01] [2026-05-21T05:38:22.526Z] ok: [an-a01n01] [2026-05-21T05:38:22.526Z] ok: [an-a01n02] [2026-05-21T05:38:22.526Z] ok: [an-striker02] [2026-05-21T05:38:22.526Z] [2026-05-21T05:38:22.526Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-05-21T05:38:22.526Z] if [ "$device" = "" ]; then [2026-05-21T05:38:22.526Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-05-21T05:38:22.526Z] exit 1 [2026-05-21T05:38:22.526Z] fi [2026-05-21T05:38:22.526Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-05-21T05:38:22.526Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-05-21T05:38:22.526Z] exit 1 [2026-05-21T05:38:22.526Z] fi [2026-05-21T05:38:22.526Z] exit 0 [2026-05-21T05:38:22.526Z] ] *** [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=bcn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=bcn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=bcn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=ifn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=sn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=sn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-05-21T05:38:31.712Z] skipping: [an-a01n02] => (item=lo) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=ifn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=bcn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=ifn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=ifn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=bcn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=ifn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-05-21T05:38:31.712Z] skipping: [an-striker01] => (item=lo) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=sn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=sn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=bcn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=sn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=ifn1_link1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=sn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=bcn1_bridge1) [2026-05-21T05:38:31.712Z] skipping: [an-a01dr01] => (item=lo) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=sn1_link2) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=bcn1_link1) [2026-05-21T05:38:31.712Z] skipping: [an-striker02] => (item=lo) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=ifn1_bridge1) [2026-05-21T05:38:31.712Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-05-21T05:38:31.712Z] skipping: [an-a01n01] => (item=lo) [2026-05-21T05:38:31.712Z] changed: [an-a01dr01] => (item=ifn1_bond1) [2026-05-21T05:38:31.712Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-05-21T05:38:31.712Z] [2026-05-21T05:38:31.712Z] PLAY [check pacemaker status after deployment] ********************************* [2026-05-21T05:38:31.712Z] [2026-05-21T05:38:31.712Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:38:32.472Z] ok: [an-a01n01] [2026-05-21T05:38:32.472Z] ok: [an-a01n02] [2026-05-21T05:38:32.472Z] [2026-05-21T05:38:32.472Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-05-21T05:38:32.472Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T05:38:32.472Z] [2026-05-21T05:38:32.472Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T05:38:32.472Z] ] *** [2026-05-21T05:38:36.426Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T05:38:36.426Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T05:38:36.426Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T05:38:36.426Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T05:38:36.426Z] [2026-05-21T05:38:36.426Z] PLAY [check drbd kmod] ********************************************************* [2026-05-21T05:38:36.426Z] [2026-05-21T05:38:36.426Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:38:38.186Z] ok: [an-a01n01] [2026-05-21T05:38:38.186Z] ok: [an-a01n02] [2026-05-21T05:38:38.186Z] ok: [an-a01dr01] [2026-05-21T05:38:38.186Z] [2026-05-21T05:38:38.186Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-05-21T05:38:38.186Z] ] *** [2026-05-21T05:38:38.944Z] ok: [an-a01n01] [2026-05-21T05:38:38.944Z] ok: [an-a01n02] [2026-05-21T05:38:38.944Z] ok: [an-a01dr01] [2026-05-21T05:38:38.944Z] [2026-05-21T05:38:38.944Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-05-21T05:38:38.944Z] ] *********************** [2026-05-21T05:38:38.944Z] skipping: [an-a01n01] [2026-05-21T05:38:38.944Z] skipping: [an-a01n02] [2026-05-21T05:38:38.944Z] skipping: [an-a01dr01] [2026-05-21T05:38:38.944Z] [2026-05-21T05:38:38.944Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-05-21T05:38:38.944Z] ] *** [2026-05-21T05:38:38.944Z] skipping: [an-a01n01] [2026-05-21T05:38:39.301Z] skipping: [an-a01n02] [2026-05-21T05:38:39.301Z] skipping: [an-a01dr01] [2026-05-21T05:38:39.301Z] [2026-05-21T05:38:39.301Z] PLAY [Copy iso images to striker01] ******************************************** [2026-05-21T05:38:39.301Z] [2026-05-21T05:38:39.301Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:38:41.772Z] ok: [an-striker01] [2026-05-21T05:38:41.772Z] [2026-05-21T05:38:41.772Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-05-21T05:38:42.997Z] ok: [an-striker01] [2026-05-21T05:38:42.997Z] [2026-05-21T05:38:42.997Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-05-21T05:38:43.755Z] changed: [an-striker01] [2026-05-21T05:38:43.755Z] [2026-05-21T05:38:43.755Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-05-21T05:38:55.736Z] changed: [an-striker01] => (item=deploy1.iso) [2026-05-21T05:38:55.736Z] changed: [an-striker01] => (item=deploy2.iso) [2026-05-21T05:38:55.736Z] changed: [an-striker01] => (item=deploy3.iso) [2026-05-21T05:38:55.736Z] changed: [an-striker01] => (item=deploy4.iso) [2026-05-21T05:38:55.736Z] changed: [an-striker01] => (item=deploy5.iso) [2026-05-21T05:38:55.736Z] [2026-05-21T05:38:55.736Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-05-21T05:38:55.736Z] ] *** [2026-05-21T05:38:55.736Z] changed: [an-striker01] [2026-05-21T05:38:55.736Z] [2026-05-21T05:38:55.736Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-05-21T05:38:56.963Z] changed: [an-striker01] [2026-05-21T05:38:56.963Z] [2026-05-21T05:38:56.963Z] PLAY [Check iso images propagation] ******************************************** [2026-05-21T05:38:56.963Z] [2026-05-21T05:38:56.963Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:38:59.511Z] ok: [an-a01n02] [2026-05-21T05:38:59.511Z] ok: [an-a01n01] [2026-05-21T05:38:59.511Z] ok: [an-a01dr01] [2026-05-21T05:38:59.511Z] ok: [an-striker01] [2026-05-21T05:38:59.511Z] ok: [an-striker02] [2026-05-21T05:38:59.511Z] [2026-05-21T05:38:59.511Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-05-21T05:38:59.511Z] exit 1 [2026-05-21T05:38:59.511Z] else [2026-05-21T05:38:59.511Z] exit 0 [2026-05-21T05:38:59.511Z] fi [2026-05-21T05:38:59.511Z] ] *** [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (170 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (170 retries left). [2026-05-21T05:41:08.464Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (170 retries left). [2026-05-21T05:41:08.464Z] ok: [an-striker01] [2026-05-21T05:41:08.464Z] ok: [an-striker02] [2026-05-21T05:41:08.464Z] ok: [an-a01n01] [2026-05-21T05:41:08.464Z] ok: [an-a01dr01] [2026-05-21T05:41:08.464Z] ok: [an-a01n02] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] PLAY [example tests] *********************************************************** [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:41:08.464Z] ok: [an-a01n01] [2026-05-21T05:41:08.464Z] ok: [an-a01dr01] [2026-05-21T05:41:08.464Z] ok: [an-a01n02] [2026-05-21T05:41:08.464Z] ok: [an-striker01] [2026-05-21T05:41:08.464Z] ok: [an-striker02] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-05-21T05:41:08.464Z] ] *** [2026-05-21T05:41:08.464Z] skipping: [an-a01n01] [2026-05-21T05:41:08.464Z] skipping: [an-a01n02] [2026-05-21T05:41:08.464Z] skipping: [an-a01dr01] [2026-05-21T05:41:08.464Z] ok: [an-striker01] [2026-05-21T05:41:08.464Z] ok: [an-striker02] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-05-21T05:41:08.464Z] ] *** [2026-05-21T05:41:08.464Z] skipping: [an-striker01] [2026-05-21T05:41:08.464Z] skipping: [an-striker02] [2026-05-21T05:41:08.464Z] ok: [an-a01n01] [2026-05-21T05:41:08.464Z] ok: [an-a01n02] [2026-05-21T05:41:08.464Z] ok: [an-a01dr01] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [inject failure _raw_params=/usr/bin/false [2026-05-21T05:41:08.464Z] ] ******************************* [2026-05-21T05:41:08.464Z] skipping: [an-striker01] [2026-05-21T05:41:08.464Z] skipping: [an-striker02] [2026-05-21T05:41:08.464Z] skipping: [an-a01n01] [2026-05-21T05:41:08.464Z] skipping: [an-a01n02] [2026-05-21T05:41:08.464Z] skipping: [an-a01dr01] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] PLAY [Record anvil available resources from striker01] ************************* [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:41:08.464Z] ok: [an-striker01] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-05-21T05:41:08.464Z] ] *** [2026-05-21T05:41:08.464Z] ok: [an-striker01] [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] PLAY [check selinux status] **************************************************** [2026-05-21T05:41:08.464Z] [2026-05-21T05:41:08.464Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:41:09.713Z] ok: [an-striker01] [2026-05-21T05:41:09.713Z] ok: [an-a01n02] [2026-05-21T05:41:09.713Z] ok: [an-a01n01] [2026-05-21T05:41:09.713Z] ok: [an-a01dr01] [2026-05-21T05:41:09.713Z] ok: [an-striker02] [2026-05-21T05:41:09.713Z] [2026-05-21T05:41:09.713Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-05-21T05:41:09.713Z] skipping: [an-striker01] [2026-05-21T05:41:09.713Z] skipping: [an-striker02] [2026-05-21T05:41:09.713Z] skipping: [an-a01n01] [2026-05-21T05:41:09.713Z] skipping: [an-a01n02] [2026-05-21T05:41:09.713Z] skipping: [an-a01dr01] [2026-05-21T05:41:09.713Z] [2026-05-21T05:41:09.713Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-05-21T05:41:10.068Z] ok: [an-striker01] => { [2026-05-21T05:41:10.068Z] "ansible_selinux": { [2026-05-21T05:41:10.068Z] "config_mode": "enforcing", [2026-05-21T05:41:10.068Z] "mode": "enforcing", [2026-05-21T05:41:10.068Z] "policyvers": 33, [2026-05-21T05:41:10.068Z] "status": "enabled", [2026-05-21T05:41:10.068Z] "type": "targeted" [2026-05-21T05:41:10.068Z] }, [2026-05-21T05:41:10.068Z] "failed_when_result": false [2026-05-21T05:41:10.068Z] } [2026-05-21T05:41:10.422Z] ok: [an-striker02] => { [2026-05-21T05:41:10.422Z] "ansible_selinux": { [2026-05-21T05:41:10.422Z] "config_mode": "enforcing", [2026-05-21T05:41:10.422Z] "mode": "enforcing", [2026-05-21T05:41:10.422Z] "policyvers": 33, [2026-05-21T05:41:10.422Z] "status": "enabled", [2026-05-21T05:41:10.422Z] "type": "targeted" [2026-05-21T05:41:10.422Z] }, [2026-05-21T05:41:10.422Z] "failed_when_result": false [2026-05-21T05:41:10.422Z] } [2026-05-21T05:41:10.422Z] ok: [an-a01n01] => { [2026-05-21T05:41:10.422Z] "ansible_selinux": { [2026-05-21T05:41:10.422Z] "config_mode": "enforcing", [2026-05-21T05:41:10.422Z] "mode": "enforcing", [2026-05-21T05:41:10.422Z] "policyvers": 33, [2026-05-21T05:41:10.422Z] "status": "enabled", [2026-05-21T05:41:10.422Z] "type": "targeted" [2026-05-21T05:41:10.422Z] }, [2026-05-21T05:41:10.422Z] "failed_when_result": false [2026-05-21T05:41:10.422Z] } [2026-05-21T05:41:10.422Z] ok: [an-a01n02] => { [2026-05-21T05:41:10.422Z] "ansible_selinux": { [2026-05-21T05:41:10.422Z] "config_mode": "enforcing", [2026-05-21T05:41:10.422Z] "mode": "enforcing", [2026-05-21T05:41:10.422Z] "policyvers": 33, [2026-05-21T05:41:10.422Z] "status": "enabled", [2026-05-21T05:41:10.422Z] "type": "targeted" [2026-05-21T05:41:10.422Z] }, [2026-05-21T05:41:10.422Z] "failed_when_result": false [2026-05-21T05:41:10.422Z] } [2026-05-21T05:41:10.422Z] ok: [an-a01dr01] => { [2026-05-21T05:41:10.422Z] "ansible_selinux": { [2026-05-21T05:41:10.422Z] "config_mode": "enforcing", [2026-05-21T05:41:10.422Z] "mode": "enforcing", [2026-05-21T05:41:10.422Z] "policyvers": 33, [2026-05-21T05:41:10.422Z] "status": "enabled", [2026-05-21T05:41:10.422Z] "type": "targeted" [2026-05-21T05:41:10.422Z] }, [2026-05-21T05:41:10.422Z] "failed_when_result": false [2026-05-21T05:41:10.422Z] } [2026-05-21T05:41:10.422Z] [2026-05-21T05:41:10.422Z] PLAY [Create random file on striker01] ***************************************** [2026-05-21T05:41:10.422Z] [2026-05-21T05:41:10.422Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:41:12.856Z] ok: [an-striker01] [2026-05-21T05:41:12.856Z] [2026-05-21T05:41:12.856Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-05-21T05:41:14.080Z] changed: [an-striker01] [2026-05-21T05:41:14.080Z] [2026-05-21T05:41:14.080Z] PLAY [Check archive file propagation] ****************************************** [2026-05-21T05:41:14.080Z] [2026-05-21T05:41:14.080Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:41:17.201Z] ok: [an-a01n01] [2026-05-21T05:41:17.201Z] ok: [an-a01dr01] [2026-05-21T05:41:17.201Z] ok: [an-a01n02] [2026-05-21T05:41:17.201Z] ok: [an-striker02] [2026-05-21T05:41:17.201Z] ok: [an-striker01] [2026-05-21T05:41:17.201Z] [2026-05-21T05:41:17.201Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-05-21T05:41:17.201Z] exit 1 [2026-05-21T05:41:17.201Z] else [2026-05-21T05:41:17.201Z] exit 0 [2026-05-21T05:41:17.201Z] fi [2026-05-21T05:41:17.201Z] ] *** [2026-05-21T05:42:40.841Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-21T05:42:40.841Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-21T05:42:40.841Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-21T05:42:40.842Z] ok: [an-striker01] [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-05-21T05:42:40.842Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-05-21T05:42:40.842Z] ok: [an-striker02] [2026-05-21T05:42:40.842Z] ok: [an-a01n01] [2026-05-21T05:42:40.842Z] ok: [an-a01n02] [2026-05-21T05:42:40.842Z] ok: [an-a01dr01] [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] PLAY [test fencing] ************************************************************ [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:42:40.842Z] ok: [an-a01n01] [2026-05-21T05:42:40.842Z] ok: [an-a01n02] [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T05:42:40.842Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T05:42:40.842Z] ] *** [2026-05-21T05:42:40.842Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T05:42:40.842Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T05:42:40.842Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T05:42:40.842Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-05-21T05:42:40.842Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T05:42:40.842Z] ] ********** [2026-05-21T05:42:40.842Z] skipping: [an-a01n02] [2026-05-21T05:42:40.842Z] changed: [an-a01n01] [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T05:42:40.842Z] skipping: [an-a01n02] [2026-05-21T05:42:40.842Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-05-21T05:42:40.842Z] [2026-05-21T05:42:40.842Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T05:45:51.096Z] ok: [an-a01n01 -> localhost] [2026-05-21T05:45:51.096Z] [2026-05-21T05:45:51.096Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T05:45:51.096Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T05:45:51.096Z] [2026-05-21T05:45:51.096Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T05:45:51.096Z] ] *** [2026-05-21T05:45:52.335Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T05:45:52.335Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T05:45:52.335Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T05:45:52.335Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T05:45:52.335Z] [2026-05-21T05:45:52.335Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-05-21T05:45:52.335Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T05:45:52.335Z] [2026-05-21T05:45:52.335Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T05:45:52.335Z] ] ********** [2026-05-21T05:45:53.113Z] skipping: [an-a01n01] [2026-05-21T05:45:53.113Z] changed: [an-a01n02] [2026-05-21T05:45:53.113Z] [2026-05-21T05:45:53.113Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T05:45:53.113Z] skipping: [an-a01n01] [2026-05-21T05:45:53.113Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-05-21T05:45:53.113Z] [2026-05-21T05:45:53.113Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T05:49:17.971Z] ok: [an-a01n02 -> localhost] [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T05:49:17.971Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T05:49:17.971Z] ] *** [2026-05-21T05:49:17.971Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T05:49:17.971Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T05:49:17.971Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T05:49:17.971Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] PLAY [advanced server tests] *************************************************** [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T05:49:17.971Z] ok: [an-a01n01] [2026-05-21T05:49:17.971Z] ok: [an-a01n02] [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T05:49:17.971Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T05:49:17.971Z] ] *** [2026-05-21T05:49:17.971Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T05:49:17.971Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T05:49:17.971Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T05:49:17.971Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-21T05:49:17.971Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-21T05:49:17.971Z] [2026-05-21T05:49:17.971Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-21T05:49:17.971Z] if [ -z "$storage" ]; then [2026-05-21T05:49:17.971Z] echo "Unable to detect shared storage" [2026-05-21T05:49:17.971Z] exit 1 [2026-05-21T05:49:17.971Z] fi [2026-05-21T05:49:17.971Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T05:49:17.971Z] ] *** [2026-05-21T05:49:19.728Z] changed: [an-a01n01] [2026-05-21T05:49:19.728Z] changed: [an-a01n02] [2026-05-21T05:49:19.728Z] [2026-05-21T05:49:19.728Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-21T05:49:19.728Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-21T05:49:19.728Z] [2026-05-21T05:49:19.728Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-21T05:49:19.728Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-21T05:49:19.728Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-21T05:49:19.728Z] echo "Unable to detect all deploy iso" [2026-05-21T05:49:19.728Z] echo "Detected isos" [2026-05-21T05:49:19.728Z] echo "$isos" [2026-05-21T05:49:19.728Z] exit 1 [2026-05-21T05:49:19.728Z] fi [2026-05-21T05:49:19.728Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-21T05:49:19.728Z] if [ -z "$installiso" ]; then [2026-05-21T05:49:19.728Z] echo "Unable to detect install iso" [2026-05-21T05:49:19.728Z] echo "Detected isos" [2026-05-21T05:49:19.728Z] echo "$isos" [2026-05-21T05:49:19.728Z] exit 1 [2026-05-21T05:49:19.728Z] fi [2026-05-21T05:49:19.728Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T05:49:19.728Z] ] *** [2026-05-21T05:49:24.678Z] changed: [an-a01n01] [2026-05-21T05:49:24.678Z] changed: [an-a01n02] [2026-05-21T05:49:24.678Z] [2026-05-21T05:49:24.678Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-05-21T05:49:24.678Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T05:49:24.678Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T05:49:24.678Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T05:49:24.678Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T05:49:24.678Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T05:49:24.678Z] [2026-05-21T05:49:24.678Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T05:49:24.678Z] ] *** [2026-05-21T05:49:26.435Z] skipping: [an-a01n02] [2026-05-21T05:49:26.435Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.816893", "end": "2026-05-21 01:49:25.816402", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 01:49:24.999509", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T05:49:26.435Z] ...ignoring [2026-05-21T05:49:26.435Z] [2026-05-21T05:49:26.435Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T05:49:26.435Z] --ci-test \ [2026-05-21T05:49:26.435Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T05:49:26.435Z] --os almalinux9 \ [2026-05-21T05:49:26.435Z] --cpu 2 --ram 4G \ [2026-05-21T05:49:26.435Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T05:49:26.435Z] --install-media {{ installisouuid }} \ [2026-05-21T05:49:26.435Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T05:49:26.435Z] ] *** [2026-05-21T05:49:35.598Z] skipping: [an-a01n02] [2026-05-21T05:49:35.598Z] changed: [an-a01n01] [2026-05-21T05:49:35.598Z] [2026-05-21T05:49:35.598Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T05:49:35.598Z] skipping: [an-a01n01] [2026-05-21T05:49:35.598Z] skipping: [an-a01n02] [2026-05-21T05:49:35.598Z] [2026-05-21T05:49:35.598Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T05:49:35.598Z] skipping: [an-a01n01] [2026-05-21T05:49:35.598Z] skipping: [an-a01n02] [2026-05-21T05:49:35.598Z] [2026-05-21T05:49:35.598Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T05:49:35.598Z] skipping: [an-a01n01] [2026-05-21T05:49:35.598Z] skipping: [an-a01n02] [2026-05-21T05:49:35.598Z] [2026-05-21T05:49:35.598Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T05:49:35.598Z] skipping: [an-a01n01] [2026-05-21T05:49:35.598Z] skipping: [an-a01n02] [2026-05-21T05:49:35.598Z] [2026-05-21T05:49:35.598Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T05:49:35.598Z] ] *** [2026-05-21T05:49:35.951Z] skipping: [an-a01n02] [2026-05-21T05:49:35.951Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.849795", "end": "2026-05-21 01:49:35.745640", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 01:49:34.895845", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T05:49:35.951Z] ...ignoring [2026-05-21T05:49:35.951Z] [2026-05-21T05:49:35.951Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T05:49:35.951Z] --ci-test \ [2026-05-21T05:49:35.951Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T05:49:35.951Z] --os almalinux9 \ [2026-05-21T05:49:35.951Z] --cpu 2 --ram 4G \ [2026-05-21T05:49:35.951Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T05:49:35.951Z] --install-media {{ installisouuid }} \ [2026-05-21T05:49:35.951Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T05:49:35.951Z] ] *** [2026-05-21T05:49:45.118Z] skipping: [an-a01n02] [2026-05-21T05:49:45.118Z] changed: [an-a01n01] [2026-05-21T05:49:45.118Z] [2026-05-21T05:49:45.118Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T05:49:45.118Z] skipping: [an-a01n01] [2026-05-21T05:49:45.118Z] skipping: [an-a01n02] [2026-05-21T05:49:45.118Z] [2026-05-21T05:49:45.118Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T05:49:45.118Z] skipping: [an-a01n01] [2026-05-21T05:49:45.118Z] skipping: [an-a01n02] [2026-05-21T05:49:45.118Z] [2026-05-21T05:49:45.118Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T05:49:45.118Z] skipping: [an-a01n01] [2026-05-21T05:49:45.118Z] skipping: [an-a01n02] [2026-05-21T05:49:45.118Z] [2026-05-21T05:49:45.118Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T05:49:45.118Z] skipping: [an-a01n01] [2026-05-21T05:49:45.118Z] skipping: [an-a01n02] [2026-05-21T05:49:45.118Z] [2026-05-21T05:49:45.118Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T05:49:45.118Z] ] *** [2026-05-21T05:49:46.366Z] skipping: [an-a01n02] [2026-05-21T05:49:46.366Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.845854", "end": "2026-05-21 01:49:46.020922", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 01:49:45.175068", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T05:49:46.366Z] ...ignoring [2026-05-21T05:49:46.366Z] [2026-05-21T05:49:46.366Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T05:49:46.366Z] --ci-test \ [2026-05-21T05:49:46.366Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T05:49:46.366Z] --os almalinux9 \ [2026-05-21T05:49:46.366Z] --cpu 2 --ram 4G \ [2026-05-21T05:49:46.366Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T05:49:46.366Z] --install-media {{ installisouuid }} \ [2026-05-21T05:49:46.366Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T05:49:46.366Z] ] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] changed: [an-a01n01] [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n01] [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n01] [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n01] [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n01] [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T05:49:57.687Z] ] *** [2026-05-21T05:49:57.687Z] skipping: [an-a01n02] [2026-05-21T05:49:57.687Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.848361", "end": "2026-05-21 01:49:57.349171", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 01:49:56.500810", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T05:49:57.687Z] ...ignoring [2026-05-21T05:49:57.687Z] [2026-05-21T05:49:57.687Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T05:49:57.687Z] --ci-test \ [2026-05-21T05:49:57.687Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T05:49:57.687Z] --os almalinux9 \ [2026-05-21T05:49:57.687Z] --cpu 2 --ram 4G \ [2026-05-21T05:49:57.687Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T05:49:57.687Z] --install-media {{ installisouuid }} \ [2026-05-21T05:49:57.687Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T05:49:57.687Z] ] *** [2026-05-21T05:50:06.844Z] skipping: [an-a01n02] [2026-05-21T05:50:06.844Z] changed: [an-a01n01] [2026-05-21T05:50:06.844Z] [2026-05-21T05:50:06.844Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T05:50:06.844Z] skipping: [an-a01n01] [2026-05-21T05:50:06.844Z] skipping: [an-a01n02] [2026-05-21T05:50:06.844Z] [2026-05-21T05:50:06.844Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T05:50:06.844Z] skipping: [an-a01n01] [2026-05-21T05:50:06.844Z] skipping: [an-a01n02] [2026-05-21T05:50:06.844Z] [2026-05-21T05:50:06.844Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T05:50:06.844Z] skipping: [an-a01n01] [2026-05-21T05:50:06.844Z] skipping: [an-a01n02] [2026-05-21T05:50:06.844Z] [2026-05-21T05:50:06.844Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T05:50:06.844Z] skipping: [an-a01n01] [2026-05-21T05:50:06.844Z] skipping: [an-a01n02] [2026-05-21T05:50:06.844Z] [2026-05-21T05:50:06.844Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T05:50:06.844Z] ] *** [2026-05-21T05:50:08.065Z] skipping: [an-a01n02] [2026-05-21T05:50:08.065Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.902224", "end": "2026-05-21 01:50:07.591083", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 01:50:06.688859", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T05:50:08.065Z] ...ignoring [2026-05-21T05:50:08.065Z] [2026-05-21T05:50:08.065Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T05:50:08.065Z] --ci-test \ [2026-05-21T05:50:08.065Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T05:50:08.065Z] --os almalinux9 \ [2026-05-21T05:50:08.065Z] --cpu 2 --ram 4G \ [2026-05-21T05:50:08.065Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T05:50:08.065Z] --install-media {{ installisouuid }} \ [2026-05-21T05:50:08.065Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T05:50:08.065Z] ] *** [2026-05-21T05:50:17.231Z] skipping: [an-a01n02] [2026-05-21T05:50:17.231Z] changed: [an-a01n01] [2026-05-21T05:50:17.231Z] [2026-05-21T05:50:17.231Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T05:50:17.231Z] skipping: [an-a01n01] [2026-05-21T05:50:17.231Z] skipping: [an-a01n02] [2026-05-21T05:50:17.231Z] [2026-05-21T05:50:17.231Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T05:50:17.585Z] skipping: [an-a01n01] [2026-05-21T05:50:17.585Z] skipping: [an-a01n02] [2026-05-21T05:50:17.585Z] [2026-05-21T05:50:17.585Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T05:50:17.585Z] skipping: [an-a01n01] [2026-05-21T05:50:17.585Z] skipping: [an-a01n02] [2026-05-21T05:50:17.585Z] [2026-05-21T05:50:17.585Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T05:50:17.585Z] skipping: [an-a01n01] [2026-05-21T05:50:17.585Z] skipping: [an-a01n02] [2026-05-21T05:50:17.585Z] [2026-05-21T05:50:17.585Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-05-21T05:50:17.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T05:50:17.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T05:50:17.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T05:50:17.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T05:50:17.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T05:50:17.939Z] [2026-05-21T05:50:17.939Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T05:50:17.939Z] ] *** [2026-05-21T05:50:46.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-05-21T05:50:46.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-05-21T05:50:46.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-05-21T05:50:46.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-05-21T05:50:46.276Z] ok: [an-a01n02] [2026-05-21T05:50:46.276Z] ok: [an-a01n01] [2026-05-21T05:50:46.276Z] [2026-05-21T05:50:46.276Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T05:50:46.276Z] ] *** [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-05-21T05:52:55.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-05-21T05:52:55.245Z] ok: [an-a01n02] [2026-05-21T05:52:55.245Z] ok: [an-a01n01] [2026-05-21T05:52:55.245Z] [2026-05-21T05:52:55.245Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T05:52:55.245Z] ] *** [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-05-21T05:55:19.299Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-05-21T05:55:19.299Z] ok: [an-a01n02] [2026-05-21T05:55:19.299Z] ok: [an-a01n01] [2026-05-21T05:55:19.299Z] [2026-05-21T05:55:19.299Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T05:55:19.299Z] ] *** [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-05-21T05:58:13.744Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left). [2026-05-21T05:58:13.744Z] ok: [an-a01n02] [2026-05-21T05:58:13.744Z] ok: [an-a01n01] [2026-05-21T05:58:13.744Z] [2026-05-21T05:58:13.744Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T05:58:13.744Z] ] *** [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (344 retries left). [2026-05-21T06:01:38.244Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (344 retries left). [2026-05-21T06:01:38.244Z] ok: [an-a01n02] [2026-05-21T06:01:38.244Z] ok: [an-a01n01] [2026-05-21T06:01:38.244Z] [2026-05-21T06:01:38.244Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-05-21T06:01:38.244Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:01:38.244Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:01:38.244Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:01:38.244Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:01:38.244Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:01:38.244Z] [2026-05-21T06:01:38.244Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:01:38.244Z] ] *** [2026-05-21T06:01:39.468Z] ok: [an-a01n02] [2026-05-21T06:01:39.468Z] ok: [an-a01n01] [2026-05-21T06:01:39.468Z] [2026-05-21T06:01:39.468Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:01:39.468Z] ] *** [2026-05-21T06:01:40.692Z] ok: [an-a01n02] [2026-05-21T06:01:40.692Z] ok: [an-a01n01] [2026-05-21T06:01:40.692Z] [2026-05-21T06:01:40.692Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:01:40.692Z] ] *** [2026-05-21T06:01:41.451Z] ok: [an-a01n02] [2026-05-21T06:01:41.451Z] ok: [an-a01n01] [2026-05-21T06:01:41.451Z] [2026-05-21T06:01:41.451Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:01:41.451Z] ] *** [2026-05-21T06:01:42.674Z] ok: [an-a01n02] [2026-05-21T06:01:42.674Z] ok: [an-a01n01] [2026-05-21T06:01:42.674Z] [2026-05-21T06:01:42.674Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:01:42.674Z] ] *** [2026-05-21T06:02:31.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-05-21T06:02:31.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-05-21T06:02:31.495Z] ok: [an-a01n02] [2026-05-21T06:02:31.495Z] ok: [an-a01n01] [2026-05-21T06:02:31.495Z] [2026-05-21T06:02:31.495Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-05-21T06:02:31.495Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:02:31.495Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:02:31.495Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:02:31.495Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:02:31.495Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:02:31.495Z] [2026-05-21T06:02:31.495Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T06:02:31.495Z] ] *** [2026-05-21T06:02:31.495Z] ok: [an-a01n02] [2026-05-21T06:02:31.495Z] ok: [an-a01n01] [2026-05-21T06:02:31.495Z] [2026-05-21T06:02:31.495Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T06:02:31.495Z] ] *** [2026-05-21T06:02:31.495Z] ok: [an-a01n02] [2026-05-21T06:02:31.495Z] ok: [an-a01n01] [2026-05-21T06:02:31.495Z] [2026-05-21T06:02:31.495Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T06:02:31.495Z] ] *** [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-05-21T06:03:30.353Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-05-21T06:03:30.353Z] ok: [an-a01n02] [2026-05-21T06:03:30.353Z] ok: [an-a01n01] [2026-05-21T06:03:30.353Z] [2026-05-21T06:03:30.353Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T06:03:30.353Z] ] *** [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-05-21T06:07:09.978Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (346 retries left). [2026-05-21T06:07:09.979Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (346 retries left). [2026-05-21T06:07:09.979Z] ok: [an-a01n02] [2026-05-21T06:07:09.979Z] ok: [an-a01n01] [2026-05-21T06:07:09.979Z] [2026-05-21T06:07:09.979Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T06:07:09.979Z] ] *** [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-05-21T06:10:34.537Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-05-21T06:10:34.538Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-05-21T06:10:34.538Z] ok: [an-a01n02] [2026-05-21T06:10:34.538Z] ok: [an-a01n01] [2026-05-21T06:10:34.538Z] [2026-05-21T06:10:34.538Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-05-21T06:10:34.538Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:10:34.538Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:10:34.538Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:10:34.538Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:10:34.538Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:10:34.538Z] [2026-05-21T06:10:34.538Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:10:34.538Z] ] *** [2026-05-21T06:10:34.538Z] ok: [an-a01n02] [2026-05-21T06:10:34.538Z] ok: [an-a01n01] [2026-05-21T06:10:34.538Z] [2026-05-21T06:10:34.538Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:10:34.538Z] ] *** [2026-05-21T06:11:03.202Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-05-21T06:11:03.202Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-05-21T06:11:03.202Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-05-21T06:11:03.202Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-05-21T06:11:03.202Z] ok: [an-a01n02] [2026-05-21T06:11:03.202Z] ok: [an-a01n01] [2026-05-21T06:11:03.202Z] [2026-05-21T06:11:03.202Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:11:03.202Z] ] *** [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-05-21T06:14:28.099Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-05-21T06:14:28.100Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-05-21T06:14:28.100Z] ok: [an-a01n02] [2026-05-21T06:14:28.100Z] ok: [an-a01n01] [2026-05-21T06:14:28.100Z] [2026-05-21T06:14:28.100Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:14:28.100Z] ] *** [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-05-21T06:16:37.337Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-05-21T06:16:37.337Z] ok: [an-a01n02] [2026-05-21T06:16:37.337Z] ok: [an-a01n01] [2026-05-21T06:16:37.337Z] [2026-05-21T06:16:37.337Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:16:37.337Z] ] *** [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-05-21T06:19:01.408Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-05-21T06:19:01.408Z] ok: [an-a01n02] [2026-05-21T06:19:01.408Z] ok: [an-a01n01] [2026-05-21T06:19:01.408Z] [2026-05-21T06:19:01.408Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:19:01.408Z] [2026-05-21T06:19:01.408Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-05-21T06:19:01.408Z] ok: [an-a01n01] [2026-05-21T06:19:01.408Z] ok: [an-a01n02] [2026-05-21T06:19:01.408Z] [2026-05-21T06:19:01.408Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:19:01.408Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:19:01.408Z] [2026-05-21T06:19:01.408Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:19:01.408Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:19:01.408Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:19:01.408Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:19:01.408Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:19:01.408Z] echo "pcs:" [2026-05-21T06:19:01.408Z] echo "$pcsstatus" [2026-05-21T06:19:01.408Z] exit 1 [2026-05-21T06:19:01.408Z] fi [2026-05-21T06:19:01.408Z] exit 0 [2026-05-21T06:19:01.408Z] ] *** [2026-05-21T06:19:03.790Z] ok: [an-a01n02] [2026-05-21T06:19:03.790Z] ok: [an-a01n01] [2026-05-21T06:19:03.790Z] [2026-05-21T06:19:03.790Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:19:03.790Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:19:03.790Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:19:03.790Z] ret=0 [2026-05-21T06:19:03.790Z] if [ "$servers" != "1" ]; then [2026-05-21T06:19:03.790Z] echo "Not all servers are running on the same node!" [2026-05-21T06:19:03.790Z] echo "$pcsstatus" [2026-05-21T06:19:03.790Z] ret=1 [2026-05-21T06:19:03.790Z] fi [2026-05-21T06:19:03.790Z] echo "$hypervisor" [2026-05-21T06:19:03.790Z] exit $ret [2026-05-21T06:19:03.790Z] ] *** [2026-05-21T06:19:05.750Z] ok: [an-a01n02] [2026-05-21T06:19:05.750Z] ok: [an-a01n01] [2026-05-21T06:19:05.750Z] [2026-05-21T06:19:05.750Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-21T06:19:05.750Z] ok: [an-a01n01] [2026-05-21T06:19:05.750Z] ok: [an-a01n02] [2026-05-21T06:19:05.750Z] [2026-05-21T06:19:05.750Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-21T06:19:05.750Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:19:05.750Z] [2026-05-21T06:19:05.750Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T06:19:05.750Z] ] ********** [2026-05-21T06:19:06.510Z] skipping: [an-a01n02] [2026-05-21T06:19:06.510Z] changed: [an-a01n01] [2026-05-21T06:19:06.510Z] [2026-05-21T06:19:06.510Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T06:19:06.510Z] skipping: [an-a01n01] [2026-05-21T06:19:06.510Z] skipping: [an-a01n02] [2026-05-21T06:19:06.510Z] [2026-05-21T06:19:06.510Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-21T06:19:06.510Z] skipping: [an-a01n01] [2026-05-21T06:19:06.510Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-21T06:19:06.510Z] [2026-05-21T06:19:06.510Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-21T06:19:06.510Z] ] *** [2026-05-21T06:20:16.382Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-21T06:20:16.382Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-21T06:20:16.382Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-21T06:20:16.382Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-21T06:20:16.382Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-21T06:20:16.382Z] ok: [an-a01n02] [2026-05-21T06:20:16.382Z] [2026-05-21T06:20:16.382Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:20:16.382Z] skipping: [an-a01n01] [2026-05-21T06:20:16.382Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-21T06:20:16.382Z] [2026-05-21T06:20:16.382Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:20:16.382Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:20:16.382Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:20:16.382Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:20:16.382Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:20:16.382Z] echo "pcs:" [2026-05-21T06:20:16.382Z] echo "$pcsstatus" [2026-05-21T06:20:16.382Z] exit 1 [2026-05-21T06:20:16.382Z] fi [2026-05-21T06:20:16.382Z] exit 0 [2026-05-21T06:20:16.382Z] ] *** [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-05-21T06:21:55.753Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-05-21T06:21:55.753Z] ok: [an-a01n02] [2026-05-21T06:21:55.753Z] [2026-05-21T06:21:55.753Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:21:55.753Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:21:55.753Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:21:55.753Z] ret=0 [2026-05-21T06:21:55.753Z] if [ "$servers" != "1" ]; then [2026-05-21T06:21:55.753Z] echo "Not all servers are running on the same node!" [2026-05-21T06:21:55.753Z] echo "$pcsstatus" [2026-05-21T06:21:55.753Z] ret=1 [2026-05-21T06:21:55.753Z] fi [2026-05-21T06:21:55.753Z] echo "$hypervisor" [2026-05-21T06:21:55.753Z] exit $ret [2026-05-21T06:21:55.753Z] ] *** [2026-05-21T06:21:55.753Z] ok: [an-a01n02] [2026-05-21T06:21:55.753Z] [2026-05-21T06:21:55.753Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] => (item=1) [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] => (item=2) [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] => (item=3) [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] => (item=4) [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] => (item=5) [2026-05-21T06:21:55.753Z] skipping: [an-a01n01] [2026-05-21T06:21:55.753Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-21T06:21:55.753Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-21T06:21:55.753Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-21T06:21:55.753Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-21T06:21:55.753Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-21T06:21:55.753Z] [2026-05-21T06:21:55.753Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:21:55.753Z] ] *** [2026-05-21T06:21:58.135Z] ok: [an-a01n02] [2026-05-21T06:21:58.135Z] [2026-05-21T06:21:58.135Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:21:58.135Z] ] *** [2026-05-21T06:21:59.394Z] ok: [an-a01n02] [2026-05-21T06:21:59.394Z] [2026-05-21T06:21:59.394Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:21:59.394Z] ] *** [2026-05-21T06:22:00.152Z] ok: [an-a01n02] [2026-05-21T06:22:00.152Z] [2026-05-21T06:22:00.152Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:22:00.152Z] ] *** [2026-05-21T06:22:00.910Z] ok: [an-a01n02] [2026-05-21T06:22:00.910Z] [2026-05-21T06:22:00.910Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:22:00.910Z] ] *** [2026-05-21T06:22:01.669Z] ok: [an-a01n02] [2026-05-21T06:22:01.669Z] [2026-05-21T06:22:01.669Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-21T06:22:02.023Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:22:02.023Z] [2026-05-21T06:22:02.023Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T06:25:26.986Z] ok: [an-a01n01 -> localhost] [2026-05-21T06:25:26.986Z] ok: [an-a01n02 -> localhost] [2026-05-21T06:25:26.986Z] [2026-05-21T06:25:26.986Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:25:26.986Z] [2026-05-21T06:25:26.986Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:25:26.986Z] ] *** [2026-05-21T06:25:26.986Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:25:26.986Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:25:26.986Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:25:26.986Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:25:26.986Z] [2026-05-21T06:25:26.986Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:25:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:25:26.986Z] [2026-05-21T06:25:26.986Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:25:26.986Z] ] *** [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-05-21T06:26:51.228Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:26:51.228Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:26:51.228Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:26:51.228Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:26:51.228Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:26:51.228Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:26:51.228Z] echo "pcs:" [2026-05-21T06:26:51.228Z] echo "$pcsstatus" [2026-05-21T06:26:51.228Z] exit 1 [2026-05-21T06:26:51.228Z] fi [2026-05-21T06:26:51.228Z] exit 0 [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.228Z] ok: [an-a01n02] [2026-05-21T06:26:51.228Z] ok: [an-a01n01] [2026-05-21T06:26:51.228Z] [2026-05-21T06:26:51.228Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:26:51.228Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:26:51.228Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:26:51.228Z] ret=0 [2026-05-21T06:26:51.228Z] if [ "$servers" != "1" ]; then [2026-05-21T06:26:51.228Z] echo "Not all servers are running on the same node!" [2026-05-21T06:26:51.228Z] echo "$pcsstatus" [2026-05-21T06:26:51.228Z] ret=1 [2026-05-21T06:26:51.228Z] fi [2026-05-21T06:26:51.228Z] echo "$hypervisor" [2026-05-21T06:26:51.228Z] exit $ret [2026-05-21T06:26:51.228Z] ] *** [2026-05-21T06:26:51.607Z] ok: [an-a01n01] [2026-05-21T06:26:51.607Z] ok: [an-a01n02] [2026-05-21T06:26:51.607Z] [2026-05-21T06:26:51.607Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-21T06:26:51.607Z] ok: [an-a01n01] [2026-05-21T06:26:51.607Z] ok: [an-a01n02] [2026-05-21T06:26:51.607Z] [2026-05-21T06:26:51.607Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-21T06:26:51.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:26:51.607Z] [2026-05-21T06:26:51.607Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T06:26:51.607Z] ] ********** [2026-05-21T06:26:52.364Z] skipping: [an-a01n01] [2026-05-21T06:26:52.364Z] changed: [an-a01n02] [2026-05-21T06:26:52.364Z] [2026-05-21T06:26:52.364Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T06:26:52.364Z] skipping: [an-a01n01] [2026-05-21T06:26:52.364Z] skipping: [an-a01n02] [2026-05-21T06:26:52.364Z] [2026-05-21T06:26:52.364Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-21T06:26:52.364Z] skipping: [an-a01n02] [2026-05-21T06:26:52.364Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-05-21T06:26:52.364Z] [2026-05-21T06:26:52.364Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-21T06:26:52.364Z] ] *** [2026-05-21T06:28:02.163Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-05-21T06:28:02.163Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-05-21T06:28:02.163Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-05-21T06:28:02.163Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-05-21T06:28:02.163Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-05-21T06:28:02.163Z] ok: [an-a01n01] [2026-05-21T06:28:02.163Z] [2026-05-21T06:28:02.163Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:28:02.163Z] skipping: [an-a01n02] [2026-05-21T06:28:02.163Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-05-21T06:28:02.163Z] [2026-05-21T06:28:02.163Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:28:02.163Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:28:02.163Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:28:02.163Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:28:02.163Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:28:02.163Z] echo "pcs:" [2026-05-21T06:28:02.163Z] echo "$pcsstatus" [2026-05-21T06:28:02.163Z] exit 1 [2026-05-21T06:28:02.163Z] fi [2026-05-21T06:28:02.163Z] exit 0 [2026-05-21T06:28:02.163Z] ] *** [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-05-21T06:29:25.836Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-05-21T06:29:25.836Z] ok: [an-a01n01] [2026-05-21T06:29:25.836Z] [2026-05-21T06:29:25.836Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:29:25.836Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:29:25.836Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:29:25.836Z] ret=0 [2026-05-21T06:29:25.836Z] if [ "$servers" != "1" ]; then [2026-05-21T06:29:25.836Z] echo "Not all servers are running on the same node!" [2026-05-21T06:29:25.836Z] echo "$pcsstatus" [2026-05-21T06:29:25.836Z] ret=1 [2026-05-21T06:29:25.836Z] fi [2026-05-21T06:29:25.836Z] echo "$hypervisor" [2026-05-21T06:29:25.836Z] exit $ret [2026-05-21T06:29:25.836Z] ] *** [2026-05-21T06:29:26.594Z] ok: [an-a01n01] [2026-05-21T06:29:26.594Z] [2026-05-21T06:29:26.594Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] => (item=1) [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] => (item=2) [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] => (item=3) [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] => (item=4) [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] => (item=5) [2026-05-21T06:29:26.594Z] skipping: [an-a01n02] [2026-05-21T06:29:26.594Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-05-21T06:29:26.594Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-05-21T06:29:26.594Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-05-21T06:29:26.594Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-05-21T06:29:26.594Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-05-21T06:29:26.594Z] [2026-05-21T06:29:26.594Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:29:26.594Z] ] *** [2026-05-21T06:29:28.287Z] ok: [an-a01n01] [2026-05-21T06:29:28.287Z] [2026-05-21T06:29:28.287Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:29:28.287Z] ] *** [2026-05-21T06:29:44.505Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-05-21T06:29:44.505Z] ok: [an-a01n01] [2026-05-21T06:29:44.505Z] [2026-05-21T06:29:44.505Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:29:44.505Z] ] *** [2026-05-21T06:29:44.505Z] ok: [an-a01n01] [2026-05-21T06:29:44.505Z] [2026-05-21T06:29:44.505Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:29:44.505Z] ] *** [2026-05-21T06:29:44.505Z] ok: [an-a01n01] [2026-05-21T06:29:44.505Z] [2026-05-21T06:29:44.505Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:29:44.505Z] ] *** [2026-05-21T06:29:44.859Z] ok: [an-a01n01] [2026-05-21T06:29:44.859Z] [2026-05-21T06:29:44.859Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-21T06:29:44.859Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:29:44.859Z] [2026-05-21T06:29:44.859Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T06:33:09.353Z] ok: [an-a01n01 -> localhost] [2026-05-21T06:33:09.353Z] ok: [an-a01n02 -> localhost] [2026-05-21T06:33:09.353Z] [2026-05-21T06:33:09.353Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:33:09.353Z] [2026-05-21T06:33:09.353Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:33:09.353Z] ] *** [2026-05-21T06:33:09.353Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:33:09.353Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:33:09.353Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:33:09.353Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:33:09.353Z] [2026-05-21T06:33:09.353Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:33:09.353Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:33:09.353Z] [2026-05-21T06:33:09.353Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:33:09.353Z] ] *** [2026-05-21T06:33:22.798Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-21T06:33:22.798Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-21T06:33:22.798Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-21T06:33:22.798Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-21T06:33:22.798Z] ok: [an-a01n01] [2026-05-21T06:33:22.798Z] ok: [an-a01n02] [2026-05-21T06:33:22.798Z] [2026-05-21T06:33:22.798Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:33:22.798Z] ] *** [2026-05-21T06:33:23.556Z] ok: [an-a01n02] [2026-05-21T06:33:23.556Z] ok: [an-a01n01] [2026-05-21T06:33:23.556Z] [2026-05-21T06:33:23.556Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:33:23.556Z] ] *** [2026-05-21T06:33:24.314Z] ok: [an-a01n01] [2026-05-21T06:33:24.314Z] ok: [an-a01n02] [2026-05-21T06:33:24.314Z] [2026-05-21T06:33:24.314Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:33:24.314Z] ] *** [2026-05-21T06:33:24.669Z] ok: [an-a01n02] [2026-05-21T06:33:24.669Z] ok: [an-a01n01] [2026-05-21T06:33:24.669Z] [2026-05-21T06:33:24.669Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:33:24.669Z] ] *** [2026-05-21T06:33:25.427Z] ok: [an-a01n01] [2026-05-21T06:33:25.427Z] ok: [an-a01n02] [2026-05-21T06:33:25.427Z] [2026-05-21T06:33:25.427Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-05-21T06:33:25.427Z] ok: [an-a01n01] [2026-05-21T06:33:25.427Z] ok: [an-a01n02] [2026-05-21T06:33:25.427Z] [2026-05-21T06:33:25.427Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:33:25.781Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:33:25.781Z] [2026-05-21T06:33:25.781Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:33:25.781Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:33:25.781Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:33:25.781Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:33:25.781Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:33:25.781Z] echo "pcs:" [2026-05-21T06:33:25.781Z] echo "$pcsstatus" [2026-05-21T06:33:25.781Z] exit 1 [2026-05-21T06:33:25.781Z] fi [2026-05-21T06:33:25.781Z] exit 0 [2026-05-21T06:33:25.781Z] ] *** [2026-05-21T06:33:27.813Z] ok: [an-a01n01] [2026-05-21T06:33:27.813Z] ok: [an-a01n02] [2026-05-21T06:33:27.813Z] [2026-05-21T06:33:27.813Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:33:27.813Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:33:27.813Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:33:27.813Z] ret=0 [2026-05-21T06:33:27.813Z] if [ "$servers" != "1" ]; then [2026-05-21T06:33:27.813Z] echo "Not all servers are running on the same node!" [2026-05-21T06:33:27.813Z] echo "$pcsstatus" [2026-05-21T06:33:27.813Z] ret=1 [2026-05-21T06:33:27.813Z] fi [2026-05-21T06:33:27.813Z] echo "$hypervisor" [2026-05-21T06:33:27.813Z] exit $ret [2026-05-21T06:33:27.813Z] ] *** [2026-05-21T06:33:29.571Z] ok: [an-a01n01] [2026-05-21T06:33:29.571Z] ok: [an-a01n02] [2026-05-21T06:33:29.571Z] [2026-05-21T06:33:29.571Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-21T06:33:29.571Z] ok: [an-a01n01] [2026-05-21T06:33:29.571Z] ok: [an-a01n02] [2026-05-21T06:33:29.571Z] [2026-05-21T06:33:29.571Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-21T06:33:29.925Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:33:29.925Z] [2026-05-21T06:33:29.925Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T06:33:29.925Z] ] ********** [2026-05-21T06:33:30.683Z] skipping: [an-a01n02] [2026-05-21T06:33:30.683Z] changed: [an-a01n01] [2026-05-21T06:33:30.683Z] [2026-05-21T06:33:30.683Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T06:33:30.683Z] skipping: [an-a01n01] [2026-05-21T06:33:30.683Z] skipping: [an-a01n02] [2026-05-21T06:33:30.683Z] [2026-05-21T06:33:30.683Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-21T06:33:30.683Z] skipping: [an-a01n01] [2026-05-21T06:33:30.683Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-21T06:33:30.683Z] [2026-05-21T06:33:30.683Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-21T06:33:30.683Z] ] *** [2026-05-21T06:34:40.466Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-21T06:34:40.466Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-21T06:34:40.466Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-21T06:34:40.466Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-21T06:34:40.466Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-21T06:34:40.466Z] ok: [an-a01n02] [2026-05-21T06:34:40.466Z] [2026-05-21T06:34:40.466Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:34:40.466Z] skipping: [an-a01n01] [2026-05-21T06:34:40.466Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-21T06:34:40.466Z] [2026-05-21T06:34:40.466Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:34:40.466Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:34:40.466Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:34:40.466Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:34:40.466Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:34:40.466Z] echo "pcs:" [2026-05-21T06:34:40.466Z] echo "$pcsstatus" [2026-05-21T06:34:40.466Z] exit 1 [2026-05-21T06:34:40.466Z] fi [2026-05-21T06:34:40.466Z] exit 0 [2026-05-21T06:34:40.466Z] ] *** [2026-05-21T06:35:38.701Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-21T06:35:38.701Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-21T06:35:38.701Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-21T06:35:38.701Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-21T06:35:38.701Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-21T06:35:38.701Z] ok: [an-a01n02] [2026-05-21T06:35:38.701Z] [2026-05-21T06:35:38.701Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:35:38.701Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:35:38.701Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:35:38.701Z] ret=0 [2026-05-21T06:35:38.701Z] if [ "$servers" != "1" ]; then [2026-05-21T06:35:38.701Z] echo "Not all servers are running on the same node!" [2026-05-21T06:35:38.701Z] echo "$pcsstatus" [2026-05-21T06:35:38.701Z] ret=1 [2026-05-21T06:35:38.701Z] fi [2026-05-21T06:35:38.701Z] echo "$hypervisor" [2026-05-21T06:35:38.701Z] exit $ret [2026-05-21T06:35:38.701Z] ] *** [2026-05-21T06:35:39.460Z] ok: [an-a01n02] [2026-05-21T06:35:39.460Z] [2026-05-21T06:35:39.460Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] => (item=1) [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] => (item=2) [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] => (item=3) [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] => (item=4) [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] => (item=5) [2026-05-21T06:35:39.460Z] skipping: [an-a01n01] [2026-05-21T06:35:39.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-21T06:35:39.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-21T06:35:39.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-21T06:35:39.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-21T06:35:39.460Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-21T06:35:39.460Z] [2026-05-21T06:35:39.460Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:35:39.460Z] ] *** [2026-05-21T06:35:55.679Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-21T06:35:55.679Z] ok: [an-a01n02] [2026-05-21T06:35:55.679Z] [2026-05-21T06:35:55.679Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:35:55.679Z] ] *** [2026-05-21T06:35:55.679Z] ok: [an-a01n02] [2026-05-21T06:35:55.679Z] [2026-05-21T06:35:55.679Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:35:55.679Z] ] *** [2026-05-21T06:35:55.679Z] ok: [an-a01n02] [2026-05-21T06:35:55.679Z] [2026-05-21T06:35:55.679Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:35:55.679Z] ] *** [2026-05-21T06:35:56.437Z] ok: [an-a01n02] [2026-05-21T06:35:56.437Z] [2026-05-21T06:35:56.437Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:35:56.437Z] ] *** [2026-05-21T06:35:56.965Z] ok: [an-a01n02] [2026-05-21T06:35:56.965Z] [2026-05-21T06:35:56.965Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-21T06:35:57.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:35:57.319Z] [2026-05-21T06:35:57.319Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01 -> localhost] [2026-05-21T06:39:21.809Z] ok: [an-a01n02 -> localhost] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:39:21.809Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:39:21.809Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:39:21.809Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:39:21.809Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:39:21.809Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:39:21.809Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:39:21.809Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:39:21.809Z] echo "pcs:" [2026-05-21T06:39:21.809Z] echo "$pcsstatus" [2026-05-21T06:39:21.809Z] exit 1 [2026-05-21T06:39:21.809Z] fi [2026-05-21T06:39:21.809Z] exit 0 [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:39:21.809Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:39:21.809Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:39:21.809Z] ret=0 [2026-05-21T06:39:21.809Z] if [ "$servers" != "1" ]; then [2026-05-21T06:39:21.809Z] echo "Not all servers are running on the same node!" [2026-05-21T06:39:21.809Z] echo "$pcsstatus" [2026-05-21T06:39:21.809Z] ret=1 [2026-05-21T06:39:21.809Z] fi [2026-05-21T06:39:21.809Z] echo "$hypervisor" [2026-05-21T06:39:21.809Z] exit $ret [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-21T06:39:21.809Z] ok: [an-a01n01] [2026-05-21T06:39:21.809Z] ok: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T06:39:21.809Z] ] ********** [2026-05-21T06:39:21.809Z] skipping: [an-a01n01] [2026-05-21T06:39:21.809Z] changed: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T06:39:21.809Z] skipping: [an-a01n01] [2026-05-21T06:39:21.809Z] skipping: [an-a01n02] [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-21T06:39:21.809Z] skipping: [an-a01n02] [2026-05-21T06:39:21.809Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-05-21T06:39:21.809Z] [2026-05-21T06:39:21.809Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-21T06:39:21.809Z] ] *** [2026-05-21T06:40:31.741Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-05-21T06:40:31.741Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-05-21T06:40:31.741Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-05-21T06:40:31.741Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-05-21T06:40:31.741Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-05-21T06:40:31.741Z] ok: [an-a01n01] [2026-05-21T06:40:31.741Z] [2026-05-21T06:40:31.741Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:40:31.741Z] skipping: [an-a01n02] [2026-05-21T06:40:31.741Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-05-21T06:40:31.741Z] [2026-05-21T06:40:31.741Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:40:31.741Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:40:31.741Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:40:31.741Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:40:31.741Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:40:31.741Z] echo "pcs:" [2026-05-21T06:40:31.741Z] echo "$pcsstatus" [2026-05-21T06:40:31.741Z] exit 1 [2026-05-21T06:40:31.741Z] fi [2026-05-21T06:40:31.741Z] exit 0 [2026-05-21T06:40:31.741Z] ] *** [2026-05-21T06:41:29.976Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-21T06:41:29.976Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-21T06:41:29.976Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-21T06:41:29.976Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-21T06:41:29.976Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-21T06:41:29.976Z] ok: [an-a01n01] [2026-05-21T06:41:29.976Z] [2026-05-21T06:41:29.976Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:41:29.976Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:41:29.976Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:41:29.976Z] ret=0 [2026-05-21T06:41:29.976Z] if [ "$servers" != "1" ]; then [2026-05-21T06:41:29.976Z] echo "Not all servers are running on the same node!" [2026-05-21T06:41:29.976Z] echo "$pcsstatus" [2026-05-21T06:41:29.976Z] ret=1 [2026-05-21T06:41:29.976Z] fi [2026-05-21T06:41:29.976Z] echo "$hypervisor" [2026-05-21T06:41:29.976Z] exit $ret [2026-05-21T06:41:29.976Z] ] *** [2026-05-21T06:41:29.976Z] ok: [an-a01n01] [2026-05-21T06:41:29.976Z] [2026-05-21T06:41:29.976Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] => (item=1) [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] => (item=2) [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] => (item=3) [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] => (item=4) [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] => (item=5) [2026-05-21T06:41:29.976Z] skipping: [an-a01n02] [2026-05-21T06:41:29.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-05-21T06:41:29.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-05-21T06:41:29.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-05-21T06:41:29.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-05-21T06:41:29.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-05-21T06:41:29.976Z] [2026-05-21T06:41:29.976Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:41:29.976Z] ] *** [2026-05-21T06:41:46.195Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-21T06:41:46.195Z] ok: [an-a01n01] [2026-05-21T06:41:46.195Z] [2026-05-21T06:41:46.195Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:41:46.195Z] ] *** [2026-05-21T06:41:46.195Z] ok: [an-a01n01] [2026-05-21T06:41:46.195Z] [2026-05-21T06:41:46.195Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:41:46.195Z] ] *** [2026-05-21T06:41:46.195Z] ok: [an-a01n01] [2026-05-21T06:41:46.195Z] [2026-05-21T06:41:46.195Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:41:46.195Z] ] *** [2026-05-21T06:41:46.195Z] ok: [an-a01n01] [2026-05-21T06:41:46.195Z] [2026-05-21T06:41:46.195Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:41:46.195Z] ] *** [2026-05-21T06:41:46.953Z] ok: [an-a01n01] [2026-05-21T06:41:46.953Z] [2026-05-21T06:41:46.953Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-21T06:41:46.953Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:41:46.953Z] [2026-05-21T06:41:46.953Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n01 -> localhost] [2026-05-21T06:45:11.615Z] ok: [an-a01n02 -> localhost] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:45:11.615Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:45:11.615Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:45:11.615Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:45:11.615Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:45:11.615Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:45:11.615Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:45:11.615Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:45:11.615Z] echo "pcs:" [2026-05-21T06:45:11.615Z] echo "$pcsstatus" [2026-05-21T06:45:11.615Z] exit 1 [2026-05-21T06:45:11.615Z] fi [2026-05-21T06:45:11.615Z] exit 0 [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:45:11.615Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:45:11.615Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:45:11.615Z] ret=0 [2026-05-21T06:45:11.615Z] if [ "$servers" != "1" ]; then [2026-05-21T06:45:11.615Z] echo "Not all servers are running on the same node!" [2026-05-21T06:45:11.615Z] echo "$pcsstatus" [2026-05-21T06:45:11.615Z] ret=1 [2026-05-21T06:45:11.615Z] fi [2026-05-21T06:45:11.615Z] echo "$hypervisor" [2026-05-21T06:45:11.615Z] exit $ret [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-21T06:45:11.615Z] ok: [an-a01n01] [2026-05-21T06:45:11.615Z] ok: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-21T06:45:11.615Z] ] ********** [2026-05-21T06:45:11.615Z] skipping: [an-a01n02] [2026-05-21T06:45:11.615Z] changed: [an-a01n01] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-21T06:45:11.615Z] skipping: [an-a01n01] [2026-05-21T06:45:11.615Z] skipping: [an-a01n02] [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-21T06:45:11.615Z] skipping: [an-a01n01] [2026-05-21T06:45:11.615Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-21T06:45:11.615Z] [2026-05-21T06:45:11.615Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-21T06:45:11.615Z] ] *** [2026-05-21T06:46:21.531Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-21T06:46:21.531Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-21T06:46:21.531Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-21T06:46:21.531Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-21T06:46:21.531Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-21T06:46:21.531Z] ok: [an-a01n02] [2026-05-21T06:46:21.531Z] [2026-05-21T06:46:21.531Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-21T06:46:21.531Z] skipping: [an-a01n01] [2026-05-21T06:46:21.531Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-21T06:46:21.531Z] [2026-05-21T06:46:21.531Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:46:21.531Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:46:21.531Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:46:21.531Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:46:21.531Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:46:21.531Z] echo "pcs:" [2026-05-21T06:46:21.531Z] echo "$pcsstatus" [2026-05-21T06:46:21.531Z] exit 1 [2026-05-21T06:46:21.531Z] fi [2026-05-21T06:46:21.531Z] exit 0 [2026-05-21T06:46:21.531Z] ] *** [2026-05-21T06:47:19.786Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-21T06:47:19.786Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-21T06:47:19.786Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-21T06:47:19.786Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-21T06:47:19.786Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-21T06:47:19.786Z] ok: [an-a01n02] [2026-05-21T06:47:19.786Z] [2026-05-21T06:47:19.786Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:47:19.786Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:47:19.786Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:47:19.786Z] ret=0 [2026-05-21T06:47:19.786Z] if [ "$servers" != "1" ]; then [2026-05-21T06:47:19.786Z] echo "Not all servers are running on the same node!" [2026-05-21T06:47:19.786Z] echo "$pcsstatus" [2026-05-21T06:47:19.786Z] ret=1 [2026-05-21T06:47:19.786Z] fi [2026-05-21T06:47:19.786Z] echo "$hypervisor" [2026-05-21T06:47:19.786Z] exit $ret [2026-05-21T06:47:19.786Z] ] *** [2026-05-21T06:47:19.786Z] ok: [an-a01n02] [2026-05-21T06:47:19.786Z] [2026-05-21T06:47:19.786Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] => (item=1) [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] => (item=2) [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] => (item=3) [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] => (item=4) [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] => (item=5) [2026-05-21T06:47:19.786Z] skipping: [an-a01n01] [2026-05-21T06:47:19.786Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-21T06:47:19.786Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-21T06:47:19.786Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-21T06:47:19.786Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-21T06:47:19.786Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-21T06:47:19.786Z] [2026-05-21T06:47:19.786Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:47:19.786Z] ] *** [2026-05-21T06:47:36.133Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-21T06:47:36.133Z] ok: [an-a01n02] [2026-05-21T06:47:36.133Z] [2026-05-21T06:47:36.133Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:47:36.133Z] ] *** [2026-05-21T06:47:36.133Z] ok: [an-a01n02] [2026-05-21T06:47:36.133Z] [2026-05-21T06:47:36.133Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:47:36.133Z] ] *** [2026-05-21T06:47:36.133Z] ok: [an-a01n02] [2026-05-21T06:47:36.133Z] [2026-05-21T06:47:36.133Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:47:36.133Z] ] *** [2026-05-21T06:47:36.133Z] ok: [an-a01n02] [2026-05-21T06:47:36.133Z] [2026-05-21T06:47:36.133Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T06:47:36.133Z] ] *** [2026-05-21T06:47:36.890Z] ok: [an-a01n02] [2026-05-21T06:47:36.890Z] [2026-05-21T06:47:36.890Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-21T06:47:36.890Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-21T06:47:36.890Z] [2026-05-21T06:47:36.890Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T06:50:49.608Z] ok: [an-a01n01 -> localhost] [2026-05-21T06:50:49.608Z] ok: [an-a01n02 -> localhost] [2026-05-21T06:50:49.608Z] [2026-05-21T06:50:49.608Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-21T06:50:49.608Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:50:49.608Z] [2026-05-21T06:50:49.608Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:50:49.608Z] ] *** [2026-05-21T06:50:52.715Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:50:52.715Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:50:52.715Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:50:52.715Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:50:52.715Z] [2026-05-21T06:50:52.715Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:50:52.715Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T06:50:52.715Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:50:52.715Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:50:52.715Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:50:52.715Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:50:52.715Z] [2026-05-21T06:50:52.715Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:50:52.715Z] ] *** [2026-05-21T06:50:53.068Z] ok: [an-a01n02] [2026-05-21T06:50:53.068Z] ok: [an-a01n01] [2026-05-21T06:50:53.068Z] [2026-05-21T06:50:53.068Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:50:53.068Z] ] *** [2026-05-21T06:50:53.827Z] ok: [an-a01n02] [2026-05-21T06:50:53.827Z] ok: [an-a01n01] [2026-05-21T06:50:53.827Z] [2026-05-21T06:50:53.828Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:50:53.828Z] ] *** [2026-05-21T06:50:54.586Z] ok: [an-a01n02] [2026-05-21T06:50:54.586Z] ok: [an-a01n01] [2026-05-21T06:50:54.586Z] [2026-05-21T06:50:54.586Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:50:54.586Z] ] *** [2026-05-21T06:50:55.346Z] ok: [an-a01n02] [2026-05-21T06:50:55.346Z] ok: [an-a01n01] [2026-05-21T06:50:55.346Z] [2026-05-21T06:50:55.346Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:50:55.346Z] ] *** [2026-05-21T06:50:56.105Z] ok: [an-a01n01] [2026-05-21T06:50:56.105Z] ok: [an-a01n02] [2026-05-21T06:50:56.105Z] [2026-05-21T06:50:56.105Z] PLAY [basic server tests] ****************************************************** [2026-05-21T06:50:56.105Z] [2026-05-21T06:50:56.105Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T06:50:58.485Z] ok: [an-a01n01] [2026-05-21T06:50:58.485Z] ok: [an-a01n02] [2026-05-21T06:50:58.485Z] [2026-05-21T06:50:58.485Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T06:50:58.485Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:50:58.485Z] [2026-05-21T06:50:58.485Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:50:58.485Z] ] *** [2026-05-21T06:51:02.441Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:51:02.441Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:51:02.441Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:51:02.441Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:51:02.441Z] [2026-05-21T06:51:02.441Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-21T06:51:02.441Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:02.441Z] [2026-05-21T06:51:02.441Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-21T06:51:02.441Z] if [ -z "$storage" ]; then [2026-05-21T06:51:02.441Z] echo "Unable to detect shared storage" [2026-05-21T06:51:02.441Z] exit 1 [2026-05-21T06:51:02.441Z] fi [2026-05-21T06:51:02.441Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T06:51:02.441Z] ] *** [2026-05-21T06:51:08.570Z] changed: [an-a01n01] [2026-05-21T06:51:08.570Z] changed: [an-a01n02] [2026-05-21T06:51:08.570Z] [2026-05-21T06:51:08.570Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-21T06:51:08.570Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:08.570Z] [2026-05-21T06:51:08.570Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-21T06:51:08.570Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-21T06:51:08.570Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-21T06:51:08.570Z] echo "Unable to detect all deploy iso" [2026-05-21T06:51:08.570Z] echo "Detected isos" [2026-05-21T06:51:08.570Z] echo "$isos" [2026-05-21T06:51:08.570Z] exit 1 [2026-05-21T06:51:08.570Z] fi [2026-05-21T06:51:08.570Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-21T06:51:08.570Z] if [ -z "$installiso" ]; then [2026-05-21T06:51:08.570Z] echo "Unable to detect install iso" [2026-05-21T06:51:08.570Z] echo "Detected isos" [2026-05-21T06:51:08.570Z] echo "$isos" [2026-05-21T06:51:08.570Z] exit 1 [2026-05-21T06:51:08.570Z] fi [2026-05-21T06:51:08.570Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T06:51:08.570Z] ] *** [2026-05-21T06:51:13.526Z] changed: [an-a01n02] [2026-05-21T06:51:13.526Z] changed: [an-a01n01] [2026-05-21T06:51:13.526Z] [2026-05-21T06:51:13.526Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-21T06:51:13.880Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:13.880Z] [2026-05-21T06:51:13.880Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:51:13.880Z] ] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] ok: [an-a01n01] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T06:51:15.639Z] --ci-test \ [2026-05-21T06:51:15.639Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T06:51:15.639Z] --os almalinux9 \ [2026-05-21T06:51:15.639Z] --cpu 2 --ram 4G \ [2026-05-21T06:51:15.639Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T06:51:15.639Z] --install-media {{ installisouuid }} \ [2026-05-21T06:51:15.639Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T06:51:15.639Z] ] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n01] [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n01] [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n01] [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n01] [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T06:51:15.639Z] skipping: [an-a01n01] [2026-05-21T06:51:15.639Z] skipping: [an-a01n02] [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-05-21T06:51:15.639Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:15.639Z] [2026-05-21T06:51:15.639Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:51:15.639Z] ] *** [2026-05-21T06:51:17.397Z] skipping: [an-a01n01] [2026-05-21T06:51:17.397Z] ok: [an-a01n02] [2026-05-21T06:51:17.397Z] [2026-05-21T06:51:17.397Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T06:51:17.397Z] --ci-test \ [2026-05-21T06:51:17.397Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T06:51:17.397Z] --os almalinux9 \ [2026-05-21T06:51:17.397Z] --cpu 2 --ram 4G \ [2026-05-21T06:51:17.397Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T06:51:17.397Z] --install-media {{ installisouuid }} \ [2026-05-21T06:51:17.397Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T06:51:17.397Z] ] *** [2026-05-21T06:51:17.397Z] skipping: [an-a01n01] [2026-05-21T06:51:17.397Z] skipping: [an-a01n02] [2026-05-21T06:51:17.397Z] [2026-05-21T06:51:17.397Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T06:51:17.397Z] skipping: [an-a01n01] [2026-05-21T06:51:17.397Z] skipping: [an-a01n02] [2026-05-21T06:51:17.397Z] [2026-05-21T06:51:17.397Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:51:17.397Z] skipping: [an-a01n01] [2026-05-21T06:51:17.397Z] skipping: [an-a01n02] [2026-05-21T06:51:17.397Z] [2026-05-21T06:51:17.397Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T06:51:17.397Z] skipping: [an-a01n01] [2026-05-21T06:51:17.397Z] skipping: [an-a01n02] [2026-05-21T06:51:17.397Z] [2026-05-21T06:51:17.397Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T06:51:17.751Z] skipping: [an-a01n01] [2026-05-21T06:51:17.751Z] skipping: [an-a01n02] [2026-05-21T06:51:17.751Z] [2026-05-21T06:51:17.751Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-21T06:51:17.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:17.751Z] [2026-05-21T06:51:17.751Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:51:17.751Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:51:17.751Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:51:17.751Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:51:17.751Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:51:17.751Z] echo "pcs:" [2026-05-21T06:51:17.751Z] echo "$pcsstatus" [2026-05-21T06:51:17.751Z] exit 1 [2026-05-21T06:51:17.751Z] fi [2026-05-21T06:51:17.751Z] exit 0 [2026-05-21T06:51:17.751Z] ] *** [2026-05-21T06:51:20.133Z] ok: [an-a01n02] [2026-05-21T06:51:20.133Z] ok: [an-a01n01] [2026-05-21T06:51:20.133Z] [2026-05-21T06:51:20.133Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:51:20.133Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:51:20.133Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:51:20.133Z] ret=0 [2026-05-21T06:51:20.133Z] if [ "$servers" != "1" ]; then [2026-05-21T06:51:20.133Z] echo "Not all servers are running on the same node!" [2026-05-21T06:51:20.133Z] echo "$pcsstatus" [2026-05-21T06:51:20.133Z] ret=1 [2026-05-21T06:51:20.133Z] fi [2026-05-21T06:51:20.133Z] echo "$hypervisor" [2026-05-21T06:51:20.133Z] exit $ret [2026-05-21T06:51:20.133Z] ] *** [2026-05-21T06:51:21.890Z] ok: [an-a01n01] [2026-05-21T06:51:21.890Z] ok: [an-a01n02] [2026-05-21T06:51:21.890Z] [2026-05-21T06:51:21.890Z] PLAY [delete extra server tests] *********************************************** [2026-05-21T06:51:21.890Z] [2026-05-21T06:51:21.890Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T06:51:24.271Z] ok: [an-a01n01] [2026-05-21T06:51:24.271Z] ok: [an-a01n02] [2026-05-21T06:51:24.271Z] [2026-05-21T06:51:24.271Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T06:51:24.271Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:51:24.271Z] [2026-05-21T06:51:24.271Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:51:24.271Z] ] *** [2026-05-21T06:51:27.775Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:51:27.775Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:51:27.775Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:51:27.775Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:51:27.775Z] [2026-05-21T06:51:27.775Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-05-21T06:51:28.129Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T06:51:28.129Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T06:51:28.129Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T06:51:28.129Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T06:51:28.129Z] [2026-05-21T06:51:28.129Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:51:28.129Z] ] *** [2026-05-21T06:51:29.886Z] skipping: [an-a01n01] [2026-05-21T06:51:29.886Z] ok: [an-a01n02] [2026-05-21T06:51:29.886Z] [2026-05-21T06:51:29.886Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T06:51:29.886Z] --force \ [2026-05-21T06:51:29.886Z] --server an-test-deploy{{ servernum }} [2026-05-21T06:51:29.886Z] ] *** [2026-05-21T06:51:37.476Z] skipping: [an-a01n01] [2026-05-21T06:51:37.476Z] changed: [an-a01n02] [2026-05-21T06:51:37.476Z] [2026-05-21T06:51:37.476Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:51:37.476Z] ] *** [2026-05-21T06:52:05.996Z] skipping: [an-a01n01] [2026-05-21T06:52:05.996Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-05-21T06:52:05.996Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-05-21T06:52:05.996Z] ok: [an-a01n02] [2026-05-21T06:52:05.996Z] [2026-05-21T06:52:05.996Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T06:52:05.996Z] ] *** [2026-05-21T06:52:05.996Z] skipping: [an-a01n01] [2026-05-21T06:52:05.996Z] ok: [an-a01n02] [2026-05-21T06:52:05.996Z] [2026-05-21T06:52:05.996Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T06:52:05.996Z] ] *** [2026-05-21T06:52:05.996Z] skipping: [an-a01n01] [2026-05-21T06:52:05.996Z] ok: [an-a01n02] [2026-05-21T06:52:05.996Z] [2026-05-21T06:52:05.996Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:52:05.996Z] ] *** [2026-05-21T06:52:05.996Z] skipping: [an-a01n01] [2026-05-21T06:52:05.996Z] ok: [an-a01n02] [2026-05-21T06:52:05.996Z] [2026-05-21T06:52:05.996Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T06:52:05.996Z] --force \ [2026-05-21T06:52:05.996Z] --server an-test-deploy{{ servernum }} [2026-05-21T06:52:05.996Z] ] *** [2026-05-21T06:52:13.615Z] skipping: [an-a01n01] [2026-05-21T06:52:13.615Z] changed: [an-a01n02] [2026-05-21T06:52:13.615Z] [2026-05-21T06:52:13.615Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:52:13.615Z] ] *** [2026-05-21T06:53:23.819Z] skipping: [an-a01n01] [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-05-21T06:53:23.819Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-05-21T06:53:23.819Z] ok: [an-a01n02] [2026-05-21T06:53:23.819Z] [2026-05-21T06:53:23.819Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T06:53:23.819Z] ] *** [2026-05-21T06:53:24.179Z] skipping: [an-a01n01] [2026-05-21T06:53:24.179Z] ok: [an-a01n02] [2026-05-21T06:53:24.179Z] [2026-05-21T06:53:24.179Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T06:53:24.179Z] ] *** [2026-05-21T06:53:30.384Z] skipping: [an-a01n01] [2026-05-21T06:53:30.384Z] ok: [an-a01n02] [2026-05-21T06:53:30.384Z] [2026-05-21T06:53:30.384Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:53:30.384Z] ] *** [2026-05-21T06:53:30.384Z] skipping: [an-a01n01] [2026-05-21T06:53:30.384Z] ok: [an-a01n02] [2026-05-21T06:53:30.384Z] [2026-05-21T06:53:30.384Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T06:53:30.384Z] --force \ [2026-05-21T06:53:30.384Z] --server an-test-deploy{{ servernum }} [2026-05-21T06:53:30.384Z] ] *** [2026-05-21T06:53:34.141Z] skipping: [an-a01n01] [2026-05-21T06:53:34.141Z] changed: [an-a01n02] [2026-05-21T06:53:34.141Z] [2026-05-21T06:53:34.141Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:53:34.141Z] ] *** [2026-05-21T06:54:33.747Z] skipping: [an-a01n01] [2026-05-21T06:54:33.747Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-05-21T06:54:33.747Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-05-21T06:54:33.747Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-05-21T06:54:33.747Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-05-21T06:54:33.747Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-05-21T06:54:33.747Z] ok: [an-a01n02] [2026-05-21T06:54:33.747Z] [2026-05-21T06:54:33.747Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T06:54:33.747Z] ] *** [2026-05-21T06:54:33.747Z] skipping: [an-a01n01] [2026-05-21T06:54:33.747Z] ok: [an-a01n02] [2026-05-21T06:54:33.747Z] [2026-05-21T06:54:33.747Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T06:54:33.747Z] ] *** [2026-05-21T06:54:34.506Z] skipping: [an-a01n01] [2026-05-21T06:54:34.506Z] ok: [an-a01n02] [2026-05-21T06:54:34.506Z] [2026-05-21T06:54:34.506Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:54:34.506Z] ] *** [2026-05-21T06:54:36.274Z] skipping: [an-a01n01] [2026-05-21T06:54:36.274Z] ok: [an-a01n02] [2026-05-21T06:54:36.274Z] [2026-05-21T06:54:36.274Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T06:54:36.274Z] --force \ [2026-05-21T06:54:36.274Z] --server an-test-deploy{{ servernum }} [2026-05-21T06:54:36.274Z] ] *** [2026-05-21T06:54:44.720Z] skipping: [an-a01n01] [2026-05-21T06:54:44.720Z] changed: [an-a01n02] [2026-05-21T06:54:44.720Z] [2026-05-21T06:54:44.720Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:54:44.720Z] ] *** [2026-05-21T06:57:39.034Z] skipping: [an-a01n01] [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (55 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (54 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (53 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (52 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (51 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (50 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (49 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (48 retries left). [2026-05-21T06:57:39.034Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (47 retries left). [2026-05-21T06:57:39.034Z] ok: [an-a01n02] [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T06:57:39.034Z] ] *** [2026-05-21T06:57:39.034Z] skipping: [an-a01n01] [2026-05-21T06:57:39.034Z] ok: [an-a01n02] [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T06:57:39.034Z] ] *** [2026-05-21T06:57:39.034Z] skipping: [an-a01n01] [2026-05-21T06:57:39.034Z] ok: [an-a01n02] [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] PLAY [create server for DR testing] ******************************************** [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T06:57:39.034Z] ok: [an-a01n02] [2026-05-21T06:57:39.034Z] ok: [an-a01n01] [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T06:57:39.034Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T06:57:39.034Z] ] *** [2026-05-21T06:57:39.034Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T06:57:39.034Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T06:57:39.034Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T06:57:39.034Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-21T06:57:39.034Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-21T06:57:39.034Z] [2026-05-21T06:57:39.034Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-21T06:57:39.034Z] if [ -z "$storage" ]; then [2026-05-21T06:57:39.034Z] echo "Unable to detect shared storage" [2026-05-21T06:57:39.034Z] exit 1 [2026-05-21T06:57:39.034Z] fi [2026-05-21T06:57:39.034Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T06:57:39.034Z] ] *** [2026-05-21T06:57:40.256Z] changed: [an-a01n01] [2026-05-21T06:57:40.256Z] changed: [an-a01n02] [2026-05-21T06:57:40.256Z] [2026-05-21T06:57:40.256Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-21T06:57:40.256Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-21T06:57:40.256Z] [2026-05-21T06:57:40.256Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-21T06:57:40.256Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-21T06:57:40.256Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-21T06:57:40.256Z] echo "Unable to detect all deploy iso" [2026-05-21T06:57:40.256Z] echo "Detected isos" [2026-05-21T06:57:40.256Z] echo "$isos" [2026-05-21T06:57:40.256Z] exit 1 [2026-05-21T06:57:40.256Z] fi [2026-05-21T06:57:40.256Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-21T06:57:40.256Z] if [ -z "$installiso" ]; then [2026-05-21T06:57:40.256Z] echo "Unable to detect install iso" [2026-05-21T06:57:40.256Z] echo "Detected isos" [2026-05-21T06:57:40.256Z] echo "$isos" [2026-05-21T06:57:40.256Z] exit 1 [2026-05-21T06:57:40.256Z] fi [2026-05-21T06:57:40.256Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T06:57:40.256Z] ] *** [2026-05-21T06:57:46.386Z] changed: [an-a01n01] [2026-05-21T06:57:46.386Z] changed: [an-a01n02] [2026-05-21T06:57:46.386Z] [2026-05-21T06:57:46.386Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-21T06:57:46.386Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-21T06:57:46.386Z] [2026-05-21T06:57:46.386Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T06:57:46.386Z] ] *** [2026-05-21T06:57:47.144Z] skipping: [an-a01n02] [2026-05-21T06:57:47.144Z] ok: [an-a01n01] [2026-05-21T06:57:47.144Z] [2026-05-21T06:57:47.144Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T06:57:47.144Z] --ci-test \ [2026-05-21T06:57:47.144Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T06:57:47.144Z] --os almalinux9 \ [2026-05-21T06:57:47.144Z] --cpu 2 --ram 4G \ [2026-05-21T06:57:47.144Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T06:57:47.144Z] --install-media {{ installisouuid }} \ [2026-05-21T06:57:47.144Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T06:57:47.144Z] ] *** [2026-05-21T06:57:47.144Z] skipping: [an-a01n01] [2026-05-21T06:57:47.144Z] skipping: [an-a01n02] [2026-05-21T06:57:47.144Z] [2026-05-21T06:57:47.144Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T06:57:47.498Z] skipping: [an-a01n01] [2026-05-21T06:57:47.498Z] skipping: [an-a01n02] [2026-05-21T06:57:47.498Z] [2026-05-21T06:57:47.498Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T06:57:47.498Z] skipping: [an-a01n01] [2026-05-21T06:57:47.498Z] skipping: [an-a01n02] [2026-05-21T06:57:47.498Z] [2026-05-21T06:57:47.498Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T06:57:47.498Z] skipping: [an-a01n01] [2026-05-21T06:57:47.498Z] skipping: [an-a01n02] [2026-05-21T06:57:47.498Z] [2026-05-21T06:57:47.498Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T06:57:47.498Z] skipping: [an-a01n01] [2026-05-21T06:57:47.498Z] skipping: [an-a01n02] [2026-05-21T06:57:47.498Z] [2026-05-21T06:57:47.498Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-21T06:57:47.498Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-21T06:57:47.498Z] [2026-05-21T06:57:47.498Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:57:47.498Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T06:57:47.498Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T06:57:47.498Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T06:57:47.498Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T06:57:47.498Z] echo "pcs:" [2026-05-21T06:57:47.498Z] echo "$pcsstatus" [2026-05-21T06:57:47.498Z] exit 1 [2026-05-21T06:57:47.498Z] fi [2026-05-21T06:57:47.498Z] exit 0 [2026-05-21T06:57:47.498Z] ] *** [2026-05-21T06:57:49.879Z] ok: [an-a01n01] [2026-05-21T06:57:49.879Z] ok: [an-a01n02] [2026-05-21T06:57:49.879Z] [2026-05-21T06:57:49.879Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T06:57:49.879Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T06:57:49.879Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T06:57:49.879Z] ret=0 [2026-05-21T06:57:49.879Z] if [ "$servers" != "1" ]; then [2026-05-21T06:57:49.879Z] echo "Not all servers are running on the same node!" [2026-05-21T06:57:49.879Z] echo "$pcsstatus" [2026-05-21T06:57:49.879Z] ret=1 [2026-05-21T06:57:49.879Z] fi [2026-05-21T06:57:49.879Z] echo "$hypervisor" [2026-05-21T06:57:49.879Z] exit $ret [2026-05-21T06:57:49.879Z] ] *** [2026-05-21T06:57:52.258Z] ok: [an-a01n01] [2026-05-21T06:57:52.258Z] ok: [an-a01n02] [2026-05-21T06:57:52.258Z] [2026-05-21T06:57:52.258Z] PLAY [DR tests] **************************************************************** [2026-05-21T06:57:52.258Z] [2026-05-21T06:57:52.258Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T06:57:54.641Z] ok: [an-a01dr01] [2026-05-21T06:57:54.641Z] [2026-05-21T06:57:54.641Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured [2026-05-21T06:57:54.641Z] ] *** [2026-05-21T06:57:58.046Z] changed: [an-a01dr01] [2026-05-21T06:57:58.046Z] [2026-05-21T06:57:58.046Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g') [2026-05-21T06:57:58.046Z] echo "DB registered hostname: $drhostname" [2026-05-21T06:57:58.046Z] echo "current hostname: $(hostname)" [2026-05-21T06:57:58.046Z] if [ "$drhostname" != "$(hostname)" ]; then [2026-05-21T06:57:58.046Z] echo "DR hostnames are not matching" [2026-05-21T06:57:58.046Z] exit 1 [2026-05-21T06:57:58.046Z] fi [2026-05-21T06:57:58.046Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then [2026-05-21T06:57:58.046Z] echo "DR hostnames are not matching drnodes an-a01dr01" [2026-05-21T06:57:58.046Z] exit 1 [2026-05-21T06:57:58.046Z] fi [2026-05-21T06:57:58.046Z] ] *** [2026-05-21T06:58:02.999Z] changed: [an-a01dr01] [2026-05-21T06:58:02.999Z] [2026-05-21T06:58:02.999Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-05-21T06:58:02.999Z] echo "Detected a linked DR host!" [2026-05-21T06:58:02.999Z] exit 1 [2026-05-21T06:58:02.999Z] fi [2026-05-21T06:58:02.999Z] ] *** [2026-05-21T06:58:06.954Z] changed: [an-a01dr01] [2026-05-21T06:58:06.954Z] [2026-05-21T06:58:06.954Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01 [2026-05-21T06:58:06.954Z] ] *** [2026-05-21T06:58:11.909Z] changed: [an-a01dr01] [2026-05-21T06:58:11.909Z] [2026-05-21T06:58:11.909Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then [2026-05-21T06:58:11.909Z] echo "DR host failed to link to anvil" [2026-05-21T06:58:11.909Z] exit 1 [2026-05-21T06:58:11.909Z] fi [2026-05-21T06:58:11.909Z] ] *** [2026-05-21T06:58:15.865Z] changed: [an-a01dr01] [2026-05-21T06:58:15.865Z] [2026-05-21T06:58:15.865Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-05-21T06:58:15.866Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-05-21T06:58:15.866Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-05-21T06:58:15.866Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-05-21T06:58:15.866Z] exit 1 [2026-05-21T06:58:15.866Z] fi [2026-05-21T06:58:15.866Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then [2026-05-21T06:58:15.866Z] echo "Unable to link local vg uuid to storage group" [2026-05-21T06:58:15.866Z] exit 1 [2026-05-21T06:58:15.866Z] fi [2026-05-21T06:58:15.866Z] ] *** [2026-05-21T06:58:23.399Z] changed: [an-a01dr01] [2026-05-21T06:58:23.399Z] [2026-05-21T06:58:23.399Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then [2026-05-21T06:58:23.399Z] echo "Local vg does not appear to be linked to storage group" [2026-05-21T06:58:23.399Z] exit 1 [2026-05-21T06:58:23.399Z] fi [2026-05-21T06:58:23.399Z] ] *** [2026-05-21T06:58:25.779Z] changed: [an-a01dr01] [2026-05-21T06:58:25.779Z] [2026-05-21T06:58:25.779Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then [2026-05-21T06:58:25.779Z] echo "Unable to add protection to server an-test-deploy1" [2026-05-21T06:58:25.779Z] exit 1 [2026-05-21T06:58:25.779Z] fi [2026-05-21T06:58:25.779Z] ] *** [2026-05-21T06:58:31.905Z] changed: [an-a01dr01] [2026-05-21T06:58:31.905Z] [2026-05-21T06:58:31.905Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-05-21T06:58:31.905Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-05-21T06:58:31.905Z] [2026-05-21T06:58:31.905Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-05-21T06:58:31.905Z] exit 1 [2026-05-21T06:58:31.905Z] fi [2026-05-21T06:58:31.905Z] ] *** [2026-05-21T06:59:20.486Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-05-21T06:59:20.486Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-05-21T06:59:20.487Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-05-21T06:59:20.487Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left). [2026-05-21T06:59:20.487Z] ok: [an-a01dr01] [2026-05-21T06:59:20.487Z] [2026-05-21T06:59:20.487Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-05-21T06:59:20.487Z] short-throw) [2026-05-21T06:59:20.487Z] expected=2 [2026-05-21T06:59:20.487Z] protocol=A [2026-05-21T06:59:20.487Z] ;; [2026-05-21T06:59:20.487Z] sync) [2026-05-21T06:59:20.487Z] expected=3 [2026-05-21T06:59:20.487Z] protocol=C [2026-05-21T06:59:20.487Z] ;; [2026-05-21T06:59:20.487Z] *) [2026-05-21T06:59:20.487Z] echo "unknown DRBD protection method" [2026-05-21T06:59:20.487Z] exit 1 [2026-05-21T06:59:20.487Z] ;; [2026-05-21T06:59:20.487Z] esac [2026-05-21T06:59:20.487Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-05-21T06:59:20.487Z] echo "Unable to confirm correct DRBD protocol" [2026-05-21T06:59:20.487Z] exit 1 [2026-05-21T06:59:20.487Z] fi [2026-05-21T06:59:20.487Z] ] *** [2026-05-21T06:59:20.487Z] changed: [an-a01dr01] [2026-05-21T06:59:20.487Z] [2026-05-21T06:59:20.487Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T06:59:20.487Z] ] *** [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left). [2026-05-21T07:01:59.652Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (107 retries left). [2026-05-21T07:01:59.652Z] ok: [an-a01dr01] [2026-05-21T07:01:59.652Z] [2026-05-21T07:01:59.652Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-05-21T07:01:59.652Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-05-21T07:01:59.652Z] [2026-05-21T07:01:59.652Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T07:01:59.652Z] ] *** [2026-05-21T07:01:59.652Z] ok: [an-a01dr01] [2026-05-21T07:01:59.653Z] [2026-05-21T07:01:59.653Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-05-21T07:01:59.653Z] echo "Unable to disconnect server from DR replication" [2026-05-21T07:01:59.653Z] exit 1 [2026-05-21T07:01:59.653Z] fi [2026-05-21T07:01:59.653Z] ] *** [2026-05-21T07:01:59.653Z] changed: [an-a01dr01] [2026-05-21T07:01:59.653Z] [2026-05-21T07:01:59.653Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:01:59.653Z] ] *** [2026-05-21T07:02:13.535Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-05-21T07:02:13.535Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-05-21T07:02:13.535Z] ok: [an-a01dr01] [2026-05-21T07:02:13.535Z] [2026-05-21T07:02:13.535Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-05-21T07:02:13.535Z] sleep 120 [2026-05-21T07:02:13.535Z] ] *** [2026-05-21T07:04:22.585Z] changed: [an-a01dr01 -> localhost] [2026-05-21T07:04:22.585Z] [2026-05-21T07:04:22.585Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-05-21T07:04:22.585Z] echo "Unable to update server DR replications" [2026-05-21T07:04:22.585Z] exit 1 [2026-05-21T07:04:22.585Z] fi [2026-05-21T07:04:22.585Z] ] *** [2026-05-21T07:04:23.346Z] changed: [an-a01dr01] [2026-05-21T07:04:23.346Z] [2026-05-21T07:04:23.346Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:04:23.346Z] ] *** [2026-05-21T07:04:46.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-05-21T07:04:46.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-05-21T07:04:46.821Z] ok: [an-a01dr01] [2026-05-21T07:04:46.821Z] [2026-05-21T07:04:46.821Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:04:46.821Z] ] *** [2026-05-21T07:05:10.330Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-05-21T07:05:10.330Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-05-21T07:05:10.330Z] ok: [an-a01dr01] [2026-05-21T07:05:10.330Z] [2026-05-21T07:05:10.330Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-05-21T07:05:10.330Z] echo "Unable to (re)connect server from DR replication" [2026-05-21T07:05:10.330Z] exit 1 [2026-05-21T07:05:10.330Z] fi [2026-05-21T07:05:10.330Z] ] *** [2026-05-21T07:05:12.743Z] changed: [an-a01dr01] [2026-05-21T07:05:12.743Z] [2026-05-21T07:05:12.743Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:05:12.743Z] ] *** [2026-05-21T07:05:46.595Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-05-21T07:05:46.595Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-05-21T07:05:46.595Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-05-21T07:05:46.595Z] ok: [an-a01dr01] [2026-05-21T07:05:46.595Z] [2026-05-21T07:05:46.595Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-05-21T07:05:46.595Z] echo "Unable to remove protection from server an-test-deploy1" [2026-05-21T07:05:46.595Z] exit 1 [2026-05-21T07:05:46.595Z] fi [2026-05-21T07:05:46.595Z] ] *** [2026-05-21T07:05:49.707Z] changed: [an-a01dr01] [2026-05-21T07:05:49.707Z] [2026-05-21T07:05:49.707Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-05-21T07:05:49.707Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-05-21T07:05:49.707Z] [2026-05-21T07:05:49.707Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:05:49.707Z] ] *** [2026-05-21T07:06:24.322Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-05-21T07:06:24.322Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-05-21T07:06:24.322Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-05-21T07:06:24.322Z] ok: [an-a01dr01] [2026-05-21T07:06:24.322Z] [2026-05-21T07:06:24.322Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:06:24.322Z] ] *** [2026-05-21T07:06:33.505Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed (60 retries left). [2026-05-21T07:06:33.505Z] ok: [an-a01dr01] [2026-05-21T07:06:33.505Z] [2026-05-21T07:06:33.505Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-05-21T07:06:33.505Z] exit 1 [2026-05-21T07:06:33.505Z] fi [2026-05-21T07:06:33.505Z] ] *** [2026-05-21T07:07:07.351Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-05-21T07:07:07.351Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-05-21T07:07:07.351Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-05-21T07:07:07.351Z] ok: [an-a01dr01] [2026-05-21T07:07:07.351Z] [2026-05-21T07:07:07.351Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-05-21T07:07:07.351Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-05-21T07:07:07.351Z] #exit 1 [2026-05-21T07:07:07.351Z] fi [2026-05-21T07:07:07.351Z] ] *** [2026-05-21T07:07:07.351Z] changed: [an-a01dr01 -> an-a01n01] [2026-05-21T07:07:07.351Z] [2026-05-21T07:07:07.351Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then [2026-05-21T07:07:07.351Z] echo "Unable to add protection to server an-test-deploy1" [2026-05-21T07:07:07.351Z] exit 1 [2026-05-21T07:07:07.351Z] fi [2026-05-21T07:07:07.351Z] ] *** [2026-05-21T07:07:12.342Z] changed: [an-a01dr01] [2026-05-21T07:07:12.342Z] [2026-05-21T07:07:12.342Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] *** [2026-05-21T07:07:12.342Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-05-21T07:07:12.342Z] [2026-05-21T07:07:12.342Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-05-21T07:07:12.342Z] exit 1 [2026-05-21T07:07:12.342Z] fi [2026-05-21T07:07:12.342Z] ] *** [2026-05-21T07:08:00.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-05-21T07:08:00.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-05-21T07:08:00.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-05-21T07:08:00.946Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left). [2026-05-21T07:08:00.946Z] ok: [an-a01dr01] [2026-05-21T07:08:00.946Z] [2026-05-21T07:08:00.946Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in [2026-05-21T07:08:00.946Z] short-throw) [2026-05-21T07:08:00.946Z] expected=2 [2026-05-21T07:08:00.946Z] protocol=A [2026-05-21T07:08:00.946Z] ;; [2026-05-21T07:08:00.946Z] sync) [2026-05-21T07:08:00.946Z] expected=3 [2026-05-21T07:08:00.946Z] protocol=C [2026-05-21T07:08:00.946Z] ;; [2026-05-21T07:08:00.946Z] *) [2026-05-21T07:08:00.946Z] echo "unknown DRBD protection method" [2026-05-21T07:08:00.946Z] exit 1 [2026-05-21T07:08:00.946Z] ;; [2026-05-21T07:08:00.946Z] esac [2026-05-21T07:08:00.946Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then [2026-05-21T07:08:00.946Z] echo "Unable to confirm correct DRBD protocol" [2026-05-21T07:08:00.946Z] exit 1 [2026-05-21T07:08:00.946Z] fi [2026-05-21T07:08:00.946Z] ] *** [2026-05-21T07:08:00.946Z] changed: [an-a01dr01] [2026-05-21T07:08:00.946Z] [2026-05-21T07:08:00.946Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T07:08:00.946Z] ] *** [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-05-21T07:10:25.167Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left). [2026-05-21T07:10:25.167Z] ok: [an-a01dr01] [2026-05-21T07:10:25.167Z] [2026-05-21T07:10:25.167Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] *** [2026-05-21T07:10:25.167Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-05-21T07:10:25.167Z] [2026-05-21T07:10:25.167Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T07:10:25.167Z] ] *** [2026-05-21T07:10:25.167Z] ok: [an-a01dr01] [2026-05-21T07:10:25.167Z] [2026-05-21T07:10:25.167Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then [2026-05-21T07:10:25.167Z] echo "Unable to disconnect server from DR replication" [2026-05-21T07:10:25.167Z] exit 1 [2026-05-21T07:10:25.167Z] fi [2026-05-21T07:10:25.167Z] ] *** [2026-05-21T07:10:25.167Z] changed: [an-a01dr01] [2026-05-21T07:10:25.167Z] [2026-05-21T07:10:25.167Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:10:25.167Z] ] *** [2026-05-21T07:10:53.741Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-05-21T07:10:53.741Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-05-21T07:10:53.741Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (118 retries left). [2026-05-21T07:10:53.741Z] ok: [an-a01dr01] [2026-05-21T07:10:53.741Z] [2026-05-21T07:10:53.741Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock" [2026-05-21T07:10:53.741Z] sleep 120 [2026-05-21T07:10:53.741Z] ] *** [2026-05-21T07:13:03.190Z] changed: [an-a01dr01 -> localhost] [2026-05-21T07:13:03.190Z] [2026-05-21T07:13:03.190Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then [2026-05-21T07:13:03.190Z] echo "Unable to update server DR replications" [2026-05-21T07:13:03.190Z] exit 1 [2026-05-21T07:13:03.190Z] fi [2026-05-21T07:13:03.190Z] ] *** [2026-05-21T07:13:04.416Z] changed: [an-a01dr01] [2026-05-21T07:13:04.416Z] [2026-05-21T07:13:04.416Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:13:04.416Z] ] *** [2026-05-21T07:13:27.978Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-05-21T07:13:27.978Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-05-21T07:13:27.978Z] ok: [an-a01dr01] [2026-05-21T07:13:27.978Z] [2026-05-21T07:13:27.978Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:13:27.978Z] ] *** [2026-05-21T07:13:39.105Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-05-21T07:13:39.105Z] ok: [an-a01dr01] [2026-05-21T07:13:39.105Z] [2026-05-21T07:13:39.105Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then [2026-05-21T07:13:39.105Z] echo "Unable to (re)connect server from DR replication" [2026-05-21T07:13:39.105Z] exit 1 [2026-05-21T07:13:39.105Z] fi [2026-05-21T07:13:39.105Z] ] *** [2026-05-21T07:13:42.840Z] changed: [an-a01dr01] [2026-05-21T07:13:42.840Z] [2026-05-21T07:13:42.840Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:13:42.840Z] ] *** [2026-05-21T07:14:06.309Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-05-21T07:14:06.309Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-05-21T07:14:06.309Z] ok: [an-a01dr01] [2026-05-21T07:14:06.309Z] [2026-05-21T07:14:06.309Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then [2026-05-21T07:14:06.309Z] echo "Unable to remove protection from server an-test-deploy1" [2026-05-21T07:14:06.310Z] exit 1 [2026-05-21T07:14:06.310Z] fi [2026-05-21T07:14:06.310Z] ] *** [2026-05-21T07:14:09.420Z] changed: [an-a01dr01] [2026-05-21T07:14:09.420Z] [2026-05-21T07:14:09.420Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] *** [2026-05-21T07:14:09.420Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-05-21T07:14:09.420Z] [2026-05-21T07:14:09.420Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:14:09.420Z] ] *** [2026-05-21T07:14:43.518Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-05-21T07:14:43.518Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-05-21T07:14:43.518Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-05-21T07:14:43.518Z] ok: [an-a01dr01] [2026-05-21T07:14:43.518Z] [2026-05-21T07:14:43.518Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:14:43.518Z] ] *** [2026-05-21T07:14:43.518Z] ok: [an-a01dr01] [2026-05-21T07:14:43.518Z] [2026-05-21T07:14:43.518Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then [2026-05-21T07:14:43.518Z] exit 1 [2026-05-21T07:14:43.518Z] fi [2026-05-21T07:14:43.518Z] ] *** [2026-05-21T07:15:17.186Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-05-21T07:15:17.186Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-05-21T07:15:17.186Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-05-21T07:15:17.186Z] ok: [an-a01dr01] [2026-05-21T07:15:17.186Z] [2026-05-21T07:15:17.186Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then [2026-05-21T07:15:17.186Z] echo "drbdadm status still reports connection to DR node (#888)" [2026-05-21T07:15:17.186Z] #exit 1 [2026-05-21T07:15:17.186Z] fi [2026-05-21T07:15:17.186Z] ] *** [2026-05-21T07:15:17.186Z] changed: [an-a01dr01 -> an-a01n01] [2026-05-21T07:15:17.186Z] [2026-05-21T07:15:17.186Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##') [2026-05-21T07:15:17.186Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}') [2026-05-21T07:15:17.186Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then [2026-05-21T07:15:17.186Z] echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]" [2026-05-21T07:15:17.186Z] exit 1 [2026-05-21T07:15:17.186Z] fi [2026-05-21T07:15:17.186Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then [2026-05-21T07:15:17.186Z] echo "Unable to unlink local vg uuid to storage group" [2026-05-21T07:15:17.186Z] exit 1 [2026-05-21T07:15:17.186Z] fi [2026-05-21T07:15:17.186Z] ] *** [2026-05-21T07:15:23.510Z] changed: [an-a01dr01] [2026-05-21T07:15:23.510Z] [2026-05-21T07:15:23.510Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then [2026-05-21T07:15:23.510Z] echo "Local vg does not appear to be unlinked from storage group (#886)" [2026-05-21T07:15:23.510Z] exit 1 [2026-05-21T07:15:23.510Z] fi [2026-05-21T07:15:23.510Z] ] *** [2026-05-21T07:15:26.160Z] changed: [an-a01dr01] [2026-05-21T07:15:26.160Z] [2026-05-21T07:15:26.160Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01 [2026-05-21T07:15:26.160Z] ] *** [2026-05-21T07:15:32.229Z] changed: [an-a01dr01] [2026-05-21T07:15:32.229Z] [2026-05-21T07:15:32.229Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "No linked DR hosts"; then [2026-05-21T07:15:32.229Z] echo "Detected a linked DR host!" [2026-05-21T07:15:32.229Z] exit 1 [2026-05-21T07:15:32.229Z] fi [2026-05-21T07:15:32.229Z] ] *** [2026-05-21T07:15:36.138Z] changed: [an-a01dr01] [2026-05-21T07:15:36.138Z] [2026-05-21T07:15:36.138Z] PLAY [upgrade server test] ***************************************************** [2026-05-21T07:15:36.138Z] [2026-05-21T07:15:36.138Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:15:37.872Z] ok: [an-a01n01] [2026-05-21T07:15:37.872Z] ok: [an-a01n02] [2026-05-21T07:15:37.872Z] [2026-05-21T07:15:37.872Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T07:15:37.872Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T07:15:37.872Z] [2026-05-21T07:15:37.872Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T07:15:37.872Z] ] *** [2026-05-21T07:15:41.783Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T07:15:41.783Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T07:15:41.783Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T07:15:41.783Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T07:15:41.783Z] [2026-05-21T07:15:41.783Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-21T07:15:41.783Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-21T07:15:41.783Z] [2026-05-21T07:15:41.783Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-21T07:15:41.783Z] if [ -z "$storage" ]; then [2026-05-21T07:15:41.783Z] echo "Unable to detect shared storage" [2026-05-21T07:15:41.783Z] exit 1 [2026-05-21T07:15:41.783Z] fi [2026-05-21T07:15:41.783Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T07:15:41.783Z] ] *** [2026-05-21T07:15:46.913Z] changed: [an-a01n01] [2026-05-21T07:15:46.913Z] changed: [an-a01n02] [2026-05-21T07:15:46.913Z] [2026-05-21T07:15:46.913Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-21T07:15:46.913Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-21T07:15:46.913Z] [2026-05-21T07:15:46.913Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-21T07:15:46.913Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-21T07:15:46.913Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-21T07:15:46.913Z] echo "Unable to detect all deploy iso" [2026-05-21T07:15:46.913Z] echo "Detected isos" [2026-05-21T07:15:46.913Z] echo "$isos" [2026-05-21T07:15:46.913Z] exit 1 [2026-05-21T07:15:46.913Z] fi [2026-05-21T07:15:46.913Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-21T07:15:46.913Z] if [ -z "$installiso" ]; then [2026-05-21T07:15:46.913Z] echo "Unable to detect install iso" [2026-05-21T07:15:46.913Z] echo "Detected isos" [2026-05-21T07:15:46.913Z] echo "$isos" [2026-05-21T07:15:46.913Z] exit 1 [2026-05-21T07:15:46.913Z] fi [2026-05-21T07:15:46.913Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-21T07:15:46.913Z] ] *** [2026-05-21T07:15:52.987Z] changed: [an-a01n01] [2026-05-21T07:15:52.987Z] changed: [an-a01n02] [2026-05-21T07:15:52.987Z] [2026-05-21T07:15:52.987Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-21T07:15:52.987Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-21T07:15:52.987Z] [2026-05-21T07:15:52.987Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:15:52.987Z] ] *** [2026-05-21T07:15:54.190Z] skipping: [an-a01n02] [2026-05-21T07:15:54.190Z] ok: [an-a01n01] [2026-05-21T07:15:54.190Z] [2026-05-21T07:15:54.190Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage cb166eba-956b-43b6-b910-92ccc534b197 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-21T07:15:54.190Z] --ci-test \ [2026-05-21T07:15:54.190Z] --name an-test-deploy{{ servernum }} \ [2026-05-21T07:15:54.190Z] --os almalinux9 \ [2026-05-21T07:15:54.190Z] --cpu 2 --ram 4G \ [2026-05-21T07:15:54.190Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-21T07:15:54.190Z] --install-media {{ installisouuid }} \ [2026-05-21T07:15:54.190Z] --driver-disc deploy{{ servernum }}.iso [2026-05-21T07:15:54.190Z] ] *** [2026-05-21T07:15:54.190Z] skipping: [an-a01n01] [2026-05-21T07:15:54.190Z] skipping: [an-a01n02] [2026-05-21T07:15:54.190Z] [2026-05-21T07:15:54.190Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-21T07:15:54.190Z] skipping: [an-a01n01] [2026-05-21T07:15:54.190Z] skipping: [an-a01n02] [2026-05-21T07:15:54.190Z] [2026-05-21T07:15:54.190Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-21T07:15:54.190Z] skipping: [an-a01n01] [2026-05-21T07:15:54.190Z] skipping: [an-a01n02] [2026-05-21T07:15:54.190Z] [2026-05-21T07:15:54.190Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-21T07:15:54.190Z] skipping: [an-a01n01] [2026-05-21T07:15:54.190Z] skipping: [an-a01n02] [2026-05-21T07:15:54.190Z] [2026-05-21T07:15:54.190Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-21T07:15:54.537Z] skipping: [an-a01n01] [2026-05-21T07:15:54.537Z] skipping: [an-a01n02] [2026-05-21T07:15:54.537Z] [2026-05-21T07:15:54.537Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-21T07:15:54.537Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-21T07:15:54.537Z] [2026-05-21T07:15:54.537Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T07:15:54.537Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-21T07:15:54.537Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-21T07:15:54.537Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-21T07:15:54.537Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-21T07:15:54.537Z] echo "pcs:" [2026-05-21T07:15:54.537Z] echo "$pcsstatus" [2026-05-21T07:15:54.537Z] exit 1 [2026-05-21T07:15:54.537Z] fi [2026-05-21T07:15:54.537Z] exit 0 [2026-05-21T07:15:54.537Z] ] *** [2026-05-21T07:15:56.270Z] ok: [an-a01n01] [2026-05-21T07:15:56.270Z] ok: [an-a01n02] [2026-05-21T07:15:56.270Z] [2026-05-21T07:15:56.270Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-21T07:15:56.270Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-21T07:15:56.270Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-21T07:15:56.270Z] ret=0 [2026-05-21T07:15:56.270Z] if [ "$servers" != "1" ]; then [2026-05-21T07:15:56.270Z] echo "Not all servers are running on the same node!" [2026-05-21T07:15:56.270Z] echo "$pcsstatus" [2026-05-21T07:15:56.270Z] ret=1 [2026-05-21T07:15:56.270Z] fi [2026-05-21T07:15:56.270Z] echo "$hypervisor" [2026-05-21T07:15:56.270Z] exit $ret [2026-05-21T07:15:56.270Z] ] *** [2026-05-21T07:15:58.617Z] ok: [an-a01n01] [2026-05-21T07:15:58.617Z] ok: [an-a01n02] [2026-05-21T07:15:58.617Z] [2026-05-21T07:15:58.618Z] PLAY [enable upgrade repo] ***************************************************** [2026-05-21T07:15:58.618Z] [2026-05-21T07:15:58.618Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:16:02.528Z] ok: [an-a01n01] [2026-05-21T07:16:02.528Z] ok: [an-a01dr01] [2026-05-21T07:16:02.528Z] ok: [an-a01n02] [2026-05-21T07:16:02.528Z] ok: [an-striker01] [2026-05-21T07:16:02.528Z] ok: [an-striker02] [2026-05-21T07:16:02.528Z] [2026-05-21T07:16:02.528Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-05-21T07:16:02.528Z] dnf config-manager --set-enabled $updaterepo [2026-05-21T07:16:02.528Z] ] *** [2026-05-21T07:16:16.079Z] changed: [an-a01dr01] [2026-05-21T07:16:16.079Z] changed: [an-a01n02] [2026-05-21T07:16:16.079Z] changed: [an-a01n01] [2026-05-21T07:16:16.079Z] changed: [an-striker01] [2026-05-21T07:16:16.079Z] changed: [an-striker02] [2026-05-21T07:16:16.079Z] [2026-05-21T07:16:16.079Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-05-21T07:16:16.079Z] ok: [an-striker01] [2026-05-21T07:16:16.079Z] ok: [an-striker02] [2026-05-21T07:16:16.079Z] ok: [an-a01n01] [2026-05-21T07:16:16.079Z] ok: [an-a01n02] [2026-05-21T07:16:16.079Z] ok: [an-a01dr01] [2026-05-21T07:16:16.079Z] [2026-05-21T07:16:16.079Z] PLAY [start upgrade process] *************************************************** [2026-05-21T07:16:16.079Z] [2026-05-21T07:16:16.079Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:16:17.810Z] ok: [an-striker01] [2026-05-21T07:16:17.810Z] [2026-05-21T07:16:17.810Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-05-21T07:16:17.810Z] ] *** [2026-05-21T07:42:51.627Z] changed: [an-striker01] [2026-05-21T07:42:51.627Z] [2026-05-21T07:42:51.627Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-05-21T07:43:19.672Z] changed: [an-striker01] [2026-05-21T07:43:19.672Z] [2026-05-21T07:43:19.672Z] PLAY [reconnect all nodes] ***************************************************** [2026-05-21T07:43:19.672Z] [2026-05-21T07:43:19.672Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:43:22.027Z] ok: [an-a01n02] [2026-05-21T07:43:22.027Z] ok: [an-a01dr01] [2026-05-21T07:43:22.027Z] ok: [an-a01n01] [2026-05-21T07:43:22.027Z] ok: [an-striker02] [2026-05-21T07:43:22.027Z] ok: [an-striker01] [2026-05-21T07:43:22.027Z] [2026-05-21T07:43:22.027Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-05-21T07:43:22.375Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01 [2026-05-21T07:43:22.375Z] [2026-05-21T07:43:22.375Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-21T07:46:46.923Z] ok: [an-striker01 -> localhost] [2026-05-21T07:46:46.923Z] ok: [an-striker02 -> localhost] [2026-05-21T07:46:46.923Z] ok: [an-a01n01 -> localhost] [2026-05-21T07:46:46.923Z] ok: [an-a01n02 -> localhost] [2026-05-21T07:46:46.923Z] ok: [an-a01dr01 -> localhost] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-05-21T07:46:46.923Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-05-21T07:46:46.923Z] exit 1 [2026-05-21T07:46:46.923Z] fi [2026-05-21T07:46:46.923Z] ] *** [2026-05-21T07:46:46.923Z] changed: [an-a01n02] [2026-05-21T07:46:46.923Z] changed: [an-a01n01] [2026-05-21T07:46:46.923Z] changed: [an-a01dr01] [2026-05-21T07:46:46.923Z] changed: [an-striker02] [2026-05-21T07:46:46.923Z] changed: [an-striker01] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] PLAY [check all nodes status after upgrade] ************************************ [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:46:46.923Z] ok: [an-a01n02] [2026-05-21T07:46:46.923Z] ok: [an-a01n01] [2026-05-21T07:46:46.923Z] ok: [an-a01dr01] [2026-05-21T07:46:46.923Z] ok: [an-striker01] [2026-05-21T07:46:46.923Z] ok: [an-striker02] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-05-21T07:46:46.923Z] echo "perl-Log-Journald update not found!" [2026-05-21T07:46:46.923Z] exit 1 [2026-05-21T07:46:46.923Z] fi [2026-05-21T07:46:46.923Z] ] *** [2026-05-21T07:46:46.923Z] changed: [an-a01n02] [2026-05-21T07:46:46.923Z] changed: [an-a01dr01] [2026-05-21T07:46:46.923Z] changed: [an-a01n01] [2026-05-21T07:46:46.923Z] changed: [an-striker01] [2026-05-21T07:46:46.923Z] changed: [an-striker02] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] PLAY [check nodes status after upgrade] **************************************** [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:46:46.923Z] ok: [an-a01n01] [2026-05-21T07:46:46.923Z] ok: [an-a01n02] [2026-05-21T07:46:46.923Z] ok: [an-a01dr01] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-05-21T07:46:46.923Z] echo "akmod-drbd update not found!" [2026-05-21T07:46:46.923Z] exit 1 [2026-05-21T07:46:46.923Z] fi [2026-05-21T07:46:46.923Z] ] *** [2026-05-21T07:46:46.923Z] changed: [an-a01n02] [2026-05-21T07:46:46.923Z] changed: [an-a01dr01] [2026-05-21T07:46:46.923Z] changed: [an-a01n01] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] PLAY [check hosted node status after upgrade] ********************************** [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:46:46.923Z] ok: [an-a01n01] [2026-05-21T07:46:46.923Z] ok: [an-a01n02] [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T07:46:46.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T07:46:46.923Z] [2026-05-21T07:46:46.923Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T07:46:46.923Z] ] *** [2026-05-21T07:46:48.128Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T07:46:48.128Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T07:46:48.128Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T07:46:48.128Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T07:46:48.128Z] [2026-05-21T07:46:48.128Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-05-21T07:46:48.475Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-05-21T07:46:48.475Z] [2026-05-21T07:46:48.475Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-21T07:46:48.475Z] ] *** [2026-05-21T07:46:50.205Z] ok: [an-a01n01] [2026-05-21T07:46:50.205Z] ok: [an-a01n02] [2026-05-21T07:46:50.205Z] [2026-05-21T07:46:50.205Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-05-21T07:46:50.205Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-05-21T07:46:50.205Z] [2026-05-21T07:46:50.205Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-21T07:46:50.205Z] ] *** [2026-05-21T07:46:50.552Z] ok: [an-a01n01] [2026-05-21T07:46:50.552Z] ok: [an-a01n02] [2026-05-21T07:46:50.552Z] [2026-05-21T07:46:50.552Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-05-21T07:46:50.899Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-05-21T07:46:50.899Z] [2026-05-21T07:46:50.899Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-21T07:46:50.899Z] ] *** [2026-05-21T07:46:51.644Z] ok: [an-a01n02] [2026-05-21T07:46:51.644Z] ok: [an-a01n01] [2026-05-21T07:46:51.644Z] [2026-05-21T07:46:51.644Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-05-21T07:46:51.644Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-05-21T07:46:51.644Z] [2026-05-21T07:46:51.644Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-21T07:46:51.644Z] ] *** [2026-05-21T07:46:52.389Z] ok: [an-a01n01] [2026-05-21T07:46:52.389Z] ok: [an-a01n02] [2026-05-21T07:46:52.389Z] [2026-05-21T07:46:52.389Z] PLAY [test anvil package self update] ****************************************** [2026-05-21T07:46:52.389Z] [2026-05-21T07:46:52.389Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:46:55.453Z] ok: [an-a01n02] [2026-05-21T07:46:55.453Z] ok: [an-a01dr01] [2026-05-21T07:46:55.453Z] ok: [an-a01n01] [2026-05-21T07:46:55.453Z] ok: [an-striker02] [2026-05-21T07:46:55.453Z] ok: [an-striker01] [2026-05-21T07:46:55.453Z] [2026-05-21T07:46:55.453Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-05-21T07:46:59.359Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-05-21T07:46:59.359Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-05-21T07:46:59.359Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-05-21T07:46:59.359Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-05-21T07:46:59.359Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-05-21T07:46:59.359Z] changed: [an-a01dr01] => (item=scancore.service) [2026-05-21T07:46:59.359Z] changed: [an-a01n02] => (item=scancore.service) [2026-05-21T07:46:59.359Z] changed: [an-a01n01] => (item=scancore.service) [2026-05-21T07:46:59.359Z] changed: [an-striker02] => (item=scancore.service) [2026-05-21T07:46:59.359Z] changed: [an-striker01] => (item=scancore.service) [2026-05-21T07:46:59.359Z] [2026-05-21T07:46:59.359Z] PLAY [test striker package self update] **************************************** [2026-05-21T07:46:59.359Z] [2026-05-21T07:46:59.359Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:47:00.559Z] ok: [an-striker01] [2026-05-21T07:47:00.559Z] ok: [an-striker02] [2026-05-21T07:47:00.559Z] [2026-05-21T07:47:00.559Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-05-21T07:47:00.559Z] ] *** [2026-05-21T07:47:28.960Z] changed: [an-striker02] [2026-05-21T07:47:28.960Z] changed: [an-striker01] [2026-05-21T07:47:28.960Z] [2026-05-21T07:47:28.960Z] PLAY [test node package self update] ******************************************* [2026-05-21T07:47:28.960Z] [2026-05-21T07:47:28.960Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:47:28.960Z] ok: [an-a01n02] [2026-05-21T07:47:28.960Z] ok: [an-a01n01] [2026-05-21T07:47:28.960Z] [2026-05-21T07:47:28.960Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-05-21T07:47:28.960Z] ] *** [2026-05-21T07:47:52.979Z] changed: [an-a01n02] [2026-05-21T07:47:52.979Z] changed: [an-a01n01] [2026-05-21T07:47:52.979Z] [2026-05-21T07:47:52.979Z] PLAY [test dr package self update] ********************************************* [2026-05-21T07:47:52.979Z] [2026-05-21T07:47:52.979Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:47:53.325Z] ok: [an-a01dr01] [2026-05-21T07:47:53.325Z] [2026-05-21T07:47:53.325Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr [2026-05-21T07:47:53.325Z] ] *** [2026-05-21T07:48:16.676Z] changed: [an-a01dr01] [2026-05-21T07:48:16.676Z] [2026-05-21T07:48:16.676Z] PLAY [test anvil package self update] ****************************************** [2026-05-21T07:48:16.676Z] [2026-05-21T07:48:16.676Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:48:17.023Z] ok: [an-striker02] [2026-05-21T07:48:17.023Z] ok: [an-a01n02] [2026-05-21T07:48:17.023Z] ok: [an-striker01] [2026-05-21T07:48:17.023Z] ok: [an-a01n01] [2026-05-21T07:48:17.023Z] ok: [an-a01dr01] [2026-05-21T07:48:17.023Z] [2026-05-21T07:48:17.023Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-05-21T07:48:19.671Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-05-21T07:48:19.671Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-05-21T07:48:19.671Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-05-21T07:48:19.671Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-05-21T07:48:19.671Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-05-21T07:48:19.671Z] changed: [an-a01n01] => (item=scancore.service) [2026-05-21T07:48:19.671Z] changed: [an-a01dr01] => (item=scancore.service) [2026-05-21T07:48:19.671Z] changed: [an-a01n02] => (item=scancore.service) [2026-05-21T07:48:19.671Z] changed: [an-striker02] => (item=scancore.service) [2026-05-21T07:48:19.671Z] changed: [an-striker01] => (item=scancore.service) [2026-05-21T07:48:19.671Z] [2026-05-21T07:48:19.671Z] PLAY [delete server tests] ***************************************************** [2026-05-21T07:48:19.671Z] [2026-05-21T07:48:19.672Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:48:21.402Z] ok: [an-a01n01] [2026-05-21T07:48:21.402Z] ok: [an-a01n02] [2026-05-21T07:48:21.402Z] [2026-05-21T07:48:21.402Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-21T07:48:21.402Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-21T07:48:21.402Z] [2026-05-21T07:48:21.402Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-21T07:48:21.402Z] ] *** [2026-05-21T07:48:26.297Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-21T07:48:26.297Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-21T07:48:26.297Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-21T07:48:26.297Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-21T07:48:26.297Z] [2026-05-21T07:48:26.297Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-05-21T07:48:26.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-21T07:48:26.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-21T07:48:26.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-21T07:48:26.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-21T07:48:26.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-21T07:48:26.297Z] [2026-05-21T07:48:26.297Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:48:26.297Z] ] *** [2026-05-21T07:48:27.499Z] skipping: [an-a01n01] [2026-05-21T07:48:27.499Z] ok: [an-a01n02] [2026-05-21T07:48:27.499Z] [2026-05-21T07:48:27.499Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T07:48:27.499Z] --force \ [2026-05-21T07:48:27.499Z] --server an-test-deploy{{ servernum }} [2026-05-21T07:48:27.499Z] ] *** [2026-05-21T07:48:34.957Z] skipping: [an-a01n01] [2026-05-21T07:48:34.957Z] changed: [an-a01n02] [2026-05-21T07:48:34.957Z] [2026-05-21T07:48:34.957Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:48:34.957Z] ] *** [2026-05-21T07:49:23.667Z] skipping: [an-a01n01] [2026-05-21T07:49:23.667Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-05-21T07:49:23.667Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-05-21T07:49:23.667Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-05-21T07:49:23.667Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-05-21T07:49:23.667Z] ok: [an-a01n02] [2026-05-21T07:49:23.667Z] [2026-05-21T07:49:23.667Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:49:23.667Z] ] *** [2026-05-21T07:49:23.667Z] skipping: [an-a01n01] [2026-05-21T07:49:23.667Z] ok: [an-a01n02] [2026-05-21T07:49:23.667Z] [2026-05-21T07:49:23.667Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:49:23.667Z] ] *** [2026-05-21T07:49:24.013Z] skipping: [an-a01n01] [2026-05-21T07:49:24.013Z] ok: [an-a01n02] [2026-05-21T07:49:24.013Z] [2026-05-21T07:49:24.013Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:24.013Z] ] *** [2026-05-21T07:49:25.744Z] skipping: [an-a01n01] [2026-05-21T07:49:25.744Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.904304", "end": "2026-05-21 03:49:25.488705", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 03:49:24.584401", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T07:49:25.744Z] ...ignoring [2026-05-21T07:49:25.744Z] [2026-05-21T07:49:25.744Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T07:49:25.744Z] --force \ [2026-05-21T07:49:25.744Z] --server an-test-deploy{{ servernum }} [2026-05-21T07:49:25.744Z] ] *** [2026-05-21T07:49:25.744Z] skipping: [an-a01n01] [2026-05-21T07:49:25.744Z] skipping: [an-a01n02] [2026-05-21T07:49:25.744Z] [2026-05-21T07:49:25.744Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:25.744Z] ] *** [2026-05-21T07:49:25.744Z] skipping: [an-a01n01] [2026-05-21T07:49:25.744Z] skipping: [an-a01n02] [2026-05-21T07:49:25.744Z] [2026-05-21T07:49:25.744Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:49:25.744Z] ] *** [2026-05-21T07:49:26.091Z] skipping: [an-a01n01] [2026-05-21T07:49:26.091Z] skipping: [an-a01n02] [2026-05-21T07:49:26.091Z] [2026-05-21T07:49:26.091Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:49:26.091Z] ] *** [2026-05-21T07:49:26.091Z] skipping: [an-a01n01] [2026-05-21T07:49:26.091Z] skipping: [an-a01n02] [2026-05-21T07:49:26.091Z] [2026-05-21T07:49:26.091Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:26.091Z] ] *** [2026-05-21T07:49:27.558Z] skipping: [an-a01n01] [2026-05-21T07:49:27.558Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.950218", "end": "2026-05-21 03:49:27.295126", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 03:49:26.344908", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T07:49:27.558Z] ...ignoring [2026-05-21T07:49:27.558Z] [2026-05-21T07:49:27.558Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T07:49:27.558Z] --force \ [2026-05-21T07:49:27.558Z] --server an-test-deploy{{ servernum }} [2026-05-21T07:49:27.558Z] ] *** [2026-05-21T07:49:27.558Z] skipping: [an-a01n01] [2026-05-21T07:49:27.558Z] skipping: [an-a01n02] [2026-05-21T07:49:27.558Z] [2026-05-21T07:49:27.558Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:27.558Z] ] *** [2026-05-21T07:49:27.558Z] skipping: [an-a01n01] [2026-05-21T07:49:27.558Z] skipping: [an-a01n02] [2026-05-21T07:49:27.558Z] [2026-05-21T07:49:27.558Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:49:27.558Z] ] *** [2026-05-21T07:49:27.904Z] skipping: [an-a01n01] [2026-05-21T07:49:27.904Z] skipping: [an-a01n02] [2026-05-21T07:49:27.904Z] [2026-05-21T07:49:27.904Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:49:27.904Z] ] *** [2026-05-21T07:49:27.904Z] skipping: [an-a01n01] [2026-05-21T07:49:27.904Z] skipping: [an-a01n02] [2026-05-21T07:49:27.904Z] [2026-05-21T07:49:27.904Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:27.904Z] ] *** [2026-05-21T07:49:29.634Z] skipping: [an-a01n01] [2026-05-21T07:49:29.634Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.935003", "end": "2026-05-21 03:49:29.181128", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 03:49:28.246125", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T07:49:29.634Z] ...ignoring [2026-05-21T07:49:29.634Z] [2026-05-21T07:49:29.634Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T07:49:29.634Z] --force \ [2026-05-21T07:49:29.634Z] --server an-test-deploy{{ servernum }} [2026-05-21T07:49:29.634Z] ] *** [2026-05-21T07:49:29.634Z] skipping: [an-a01n01] [2026-05-21T07:49:29.634Z] skipping: [an-a01n02] [2026-05-21T07:49:29.634Z] [2026-05-21T07:49:29.634Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:29.634Z] ] *** [2026-05-21T07:49:29.634Z] skipping: [an-a01n01] [2026-05-21T07:49:29.634Z] skipping: [an-a01n02] [2026-05-21T07:49:29.634Z] [2026-05-21T07:49:29.634Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:49:29.634Z] ] *** [2026-05-21T07:49:29.634Z] skipping: [an-a01n01] [2026-05-21T07:49:29.634Z] skipping: [an-a01n02] [2026-05-21T07:49:29.634Z] [2026-05-21T07:49:29.634Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:49:29.634Z] ] *** [2026-05-21T07:49:29.634Z] skipping: [an-a01n01] [2026-05-21T07:49:29.634Z] skipping: [an-a01n02] [2026-05-21T07:49:29.634Z] [2026-05-21T07:49:29.634Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:29.634Z] ] *** [2026-05-21T07:49:31.370Z] skipping: [an-a01n01] [2026-05-21T07:49:31.370Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.875723", "end": "2026-05-21 03:49:30.891610", "msg": "non-zero return code", "rc": 1, "start": "2026-05-21 03:49:30.015887", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-05-21T07:49:31.370Z] ...ignoring [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-21T07:49:31.370Z] --force \ [2026-05-21T07:49:31.370Z] --server an-test-deploy{{ servernum }} [2026-05-21T07:49:31.370Z] ] *** [2026-05-21T07:49:31.370Z] skipping: [an-a01n01] [2026-05-21T07:49:31.370Z] skipping: [an-a01n02] [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-21T07:49:31.370Z] ] *** [2026-05-21T07:49:31.370Z] skipping: [an-a01n01] [2026-05-21T07:49:31.370Z] skipping: [an-a01n02] [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-21T07:49:31.370Z] ] *** [2026-05-21T07:49:31.370Z] skipping: [an-a01n01] [2026-05-21T07:49:31.370Z] skipping: [an-a01n02] [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-21T07:49:31.370Z] ] *** [2026-05-21T07:49:31.370Z] skipping: [an-a01n01] [2026-05-21T07:49:31.370Z] skipping: [an-a01n02] [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] PLAY [Check anvil available resources from striker01] ************************** [2026-05-21T07:49:31.370Z] [2026-05-21T07:49:31.370Z] TASK [Gathering Facts ] ******************************************************** [2026-05-21T07:49:33.713Z] ok: [an-striker01] [2026-05-21T07:49:33.713Z] [2026-05-21T07:49:33.713Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-05-21T07:49:33.713Z] sleep 300 [2026-05-21T07:49:33.713Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-05-21T07:49:33.713Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-05-21T07:49:33.713Z] ] *** [2026-05-21T07:54:43.906Z] ok: [an-striker01] [2026-05-21T07:54:43.906Z] [2026-05-21T07:54:43.906Z] PLAY RECAP ********************************************************************* [2026-05-21T07:54:43.906Z] an-a01dr01 : ok=90 changed=38 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [2026-05-21T07:54:43.906Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-05-21T07:54:43.906Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-05-21T07:54:43.906Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-05-21T07:54:43.906Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-05-21T07:54:43.906Z] [2026-05-21T07:54:43.906Z] tar: Removing leading `/' from member names [2026-05-21T07:54:43.906Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-21T07:54:43.906Z] /root/ci-tools [2026-05-21T07:54:43.906Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/560.tar.gz [2026-05-21T07:54:43.906Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-05-21T07:54:43.906Z] This might take a few seconds. [2026-05-21T07:54:43.906Z] [2026-05-21T07:54:43.906Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T07:54:43.906Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T07:54:44.649Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T07:54:44.649Z] [2026-05-21T07:54:44.649Z] Manage your connected systems: https://red.ht/connector [2026-05-21T07:54:45.847Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T07:54:45.847Z] [2026-05-21T07:54:45.847Z] 1. Register with subscription-manager [2026-05-21T07:54:45.847Z] # subscription-manager register [2026-05-21T07:54:45.847Z] [2026-05-21T07:54:45.847Z] 2. Register with insights-client [2026-05-21T07:54:45.847Z] # insights-client --register [2026-05-21T07:54:47.574Z] This system is currently not registered. [2026-05-21T07:54:47.921Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-05-21T07:54:47.921Z] This might take a few seconds. [2026-05-21T07:54:47.921Z] [2026-05-21T07:54:47.921Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T07:54:50.984Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T07:54:52.183Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T07:54:52.183Z] [2026-05-21T07:54:52.183Z] Manage your connected systems: https://red.ht/connector [2026-05-21T07:54:53.909Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T07:54:53.909Z] [2026-05-21T07:54:53.909Z] 1. Register with subscription-manager [2026-05-21T07:54:53.909Z] # subscription-manager register [2026-05-21T07:54:53.909Z] [2026-05-21T07:54:53.909Z] 2. Register with insights-client [2026-05-21T07:54:53.909Z] # insights-client --register [2026-05-21T07:54:55.108Z] This system is currently not registered. [2026-05-21T07:54:55.457Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-05-21T07:54:55.457Z] This might take a few seconds. [2026-05-21T07:54:55.457Z] [2026-05-21T07:54:55.457Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T07:54:57.799Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T07:54:58.543Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T07:54:58.543Z] [2026-05-21T07:54:58.543Z] Manage your connected systems: https://red.ht/connector [2026-05-21T07:54:59.741Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T07:54:59.741Z] [2026-05-21T07:54:59.741Z] 1. Register with subscription-manager [2026-05-21T07:54:59.741Z] # subscription-manager register [2026-05-21T07:54:59.741Z] [2026-05-21T07:54:59.741Z] 2. Register with insights-client [2026-05-21T07:54:59.741Z] # insights-client --register [2026-05-21T07:55:00.941Z] This system is currently not registered. [2026-05-21T07:55:01.287Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-05-21T07:55:01.287Z] This might take a few seconds. [2026-05-21T07:55:01.287Z] [2026-05-21T07:55:01.287Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T07:55:03.016Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T07:55:03.759Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T07:55:03.759Z] [2026-05-21T07:55:03.759Z] Manage your connected systems: https://red.ht/connector [2026-05-21T07:55:04.958Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T07:55:04.958Z] [2026-05-21T07:55:04.958Z] 1. Register with subscription-manager [2026-05-21T07:55:04.958Z] # subscription-manager register [2026-05-21T07:55:04.958Z] [2026-05-21T07:55:04.958Z] 2. Register with insights-client [2026-05-21T07:55:04.958Z] # insights-client --register [2026-05-21T07:55:06.157Z] This system is currently not registered. [2026-05-21T07:55:06.503Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-05-21T07:55:06.503Z] This might take a few seconds. [2026-05-21T07:55:06.503Z] [2026-05-21T07:55:06.503Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-21T07:55:08.233Z] 𐄂 Disconnected from Red Hat Insights [2026-05-21T07:55:09.435Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-21T07:55:09.435Z] [2026-05-21T07:55:09.435Z] Manage your connected systems: https://red.ht/connector [2026-05-21T07:55:10.634Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-21T07:55:10.634Z] [2026-05-21T07:55:10.634Z] 1. Register with subscription-manager [2026-05-21T07:55:10.634Z] # subscription-manager register [2026-05-21T07:55:10.634Z] [2026-05-21T07:55:10.634Z] 2. Register with insights-client [2026-05-21T07:55:10.634Z] # insights-client --register [2026-05-21T07:55:11.378Z] This system is currently not registered. [2026-05-21T07:55:11.451Z] [Pipeline] } [2026-05-21T07:55:11.453Z] [Pipeline] // timeout [2026-05-21T07:55:11.456Z] [Pipeline] } [2026-05-21T07:55:11.458Z] [Pipeline] // script [2026-05-21T07:55:11.460Z] [Pipeline] } [2026-05-21T07:55:11.462Z] [Pipeline] // stage [2026-05-21T07:55:11.465Z] [Pipeline] stage [2026-05-21T07:55:11.465Z] [Pipeline] { (Declarative: Post Actions) [2026-05-21T07:55:11.471Z] [Pipeline] cleanWs [2026-05-21T07:55:11.522Z] [WS-CLEANUP] Deleting project workspace... [2026-05-21T07:55:11.522Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-05-21T07:55:11.547Z] [WS-CLEANUP] done [2026-05-21T07:55:11.550Z] [Pipeline] script [2026-05-21T07:55:11.553Z] [Pipeline] { [2026-05-21T07:55:11.571Z] [Pipeline] stage [2026-05-21T07:55:11.571Z] [Pipeline] { (Copying logs) [2026-05-21T07:55:11.610Z] [Pipeline] node [2026-05-21T07:55:11.716Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-05-21T07:55:11.716Z] [Pipeline] { [2026-05-21T07:55:11.758Z] [Pipeline] node [2026-05-21T07:55:11.861Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-05-21T07:55:11.862Z] [Pipeline] { [2026-05-21T07:55:11.865Z] [Pipeline] sh [2026-05-21T07:55:12.120Z] + mkdir -p /var/lib/jenkins/locks [2026-05-21T07:55:12.131Z] [Pipeline] echo [2026-05-21T07:55:12.131Z] RWLock: FD for lock log_archive in stage projectFinishUp is 394 [2026-05-21T07:55:12.134Z] [Pipeline] sleep [2026-05-21T07:55:12.134Z] No need to sleep any longer [2026-05-21T07:55:12.137Z] [Pipeline] echo [2026-05-21T07:55:12.137Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-05-21T07:55:12.140Z] [Pipeline] } [2026-05-21T07:55:12.142Z] [Pipeline] // node [2026-05-21T07:55:12.145Z] [Pipeline] echo [2026-05-21T07:55:12.145Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/560/ [2026-05-21T07:55:12.146Z] [Pipeline] sh [2026-05-21T07:55:12.402Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/560/ [2026-05-21T07:55:12.404Z] [Pipeline] sh