[2026-09-03T03:45:06.948Z] Started by upstream project "anvil/anvil-build-cron" build number 154
[2026-09-03T03:45:06.948Z] originally caused by:
[2026-09-03T03:45:06.948Z]  Started by timer
[2026-09-03T03:45:07.608Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools
[2026-09-03T03:45:07.617Z] Loading library ProjectLib@main:pipelines/projects/anvil/
[2026-09-03T03:45:07.970Z] Examining kronosnet/ci-tools
[2026-09-03T03:45:07.970Z] Attempting to resolve main as a branch
[2026-09-03T03:45:08.145Z] Resolved main as branch main at revision 2b49d76fc5c6decf37323670c17cd416dfd5a73a
[2026-09-03T03:45:08.146Z] Selected Git installation does not exist. Using Default
[2026-09-03T03:45:08.146Z] The recommended git tool is: NONE
[2026-09-03T03:45:08.146Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9
[2026-09-03T03:45:08.149Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead.
[2026-09-03T03:45:08.149Z] Wiping out workspace first.
[2026-09-03T03:45:08.174Z] Cloning the remote Git repository
[2026-09-03T03:45:08.174Z] Using shallow clone with depth 1
[2026-09-03T03:45:08.174Z] Avoid fetching tags
[2026-09-03T03:45:08.174Z] Honoring refspec on initial clone
[2026-09-03T03:45:08.175Z] Cloning repository https://github.com/kronosnet/ci-tools.git
[2026-09-03T03:45:08.175Z]  > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10
[2026-09-03T03:45:08.186Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git
[2026-09-03T03:45:08.186Z]  > git --version # timeout=10
[2026-09-03T03:45:08.193Z]  > git --version # 'git version 2.52.0'
[2026-09-03T03:45:08.193Z] using GIT_ASKPASS to set credentials getCredUUID - github
[2026-09-03T03:45:08.201Z]  > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-03T03:45:08.534Z]  > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10
[2026-09-03T03:45:08.541Z]  > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-03T03:45:08.550Z] Avoid second fetch
[2026-09-03T03:45:08.551Z] Checking out Revision 2b49d76fc5c6decf37323670c17cd416dfd5a73a (main)
[2026-09-03T03:45:08.551Z]  > git config core.sparsecheckout # timeout=10
[2026-09-03T03:45:08.559Z]  > git checkout -f 2b49d76fc5c6decf37323670c17cd416dfd5a73a # timeout=10
[2026-09-03T03:45:08.612Z] Commit message: "Fix pcs-1.x rpm filter"
[2026-09-03T03:45:08.620Z] Loading library GlobalLib@main:./pipelines/libs/global/
[2026-09-03T03:45:08.848Z] Examining kronosnet/ci-tools
[2026-09-03T03:45:08.848Z] Attempting to resolve main as a branch
[2026-09-03T03:45:09.006Z] Resolved main as branch main at revision 2b49d76fc5c6decf37323670c17cd416dfd5a73a
[2026-09-03T03:45:09.007Z] Selected Git installation does not exist. Using Default
[2026-09-03T03:45:09.007Z] The recommended git tool is: NONE
[2026-09-03T03:45:09.007Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9
[2026-09-03T03:45:09.011Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead.
[2026-09-03T03:45:09.011Z] Wiping out workspace first.
[2026-09-03T03:45:09.039Z] Cloning the remote Git repository
[2026-09-03T03:45:09.039Z] Using shallow clone with depth 1
[2026-09-03T03:45:09.039Z] Avoid fetching tags
[2026-09-03T03:45:09.039Z] Honoring refspec on initial clone
[2026-09-03T03:45:09.039Z] Cloning repository https://github.com/kronosnet/ci-tools.git
[2026-09-03T03:45:09.039Z]  > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10
[2026-09-03T03:45:09.049Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git
[2026-09-03T03:45:09.049Z]  > git --version # timeout=10
[2026-09-03T03:45:09.056Z]  > git --version # 'git version 2.52.0'
[2026-09-03T03:45:09.056Z] using GIT_ASKPASS to set credentials getCredUUID - github
[2026-09-03T03:45:09.064Z]  > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-03T03:45:09.357Z]  > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10
[2026-09-03T03:45:09.364Z]  > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-03T03:45:09.372Z] Avoid second fetch
[2026-09-03T03:45:09.373Z] Checking out Revision 2b49d76fc5c6decf37323670c17cd416dfd5a73a (main)
[2026-09-03T03:45:09.373Z]  > git config core.sparsecheckout # timeout=10
[2026-09-03T03:45:09.379Z]  > git checkout -f 2b49d76fc5c6decf37323670c17cd416dfd5a73a # timeout=10
[2026-09-03T03:45:09.413Z] Commit message: "Fix pcs-1.x rpm filter"
[2026-09-03T03:45:09.524Z] [Pipeline] Start of Pipeline
[2026-09-03T03:45:09.632Z] [Pipeline] node
[2026-09-03T03:45:09.637Z] Running on anvil-ci-bm-phy07 in /srv/workspace/anvil/anvil-build-functional-testing
[2026-09-03T03:45:09.637Z] [Pipeline] {
[2026-09-03T03:45:09.649Z] [Pipeline] stage
[2026-09-03T03:45:09.650Z] [Pipeline] { (Prep build env)
[2026-09-03T03:45:09.660Z] [Pipeline] script
[2026-09-03T03:45:09.662Z] [Pipeline] {
[2026-09-03T03:45:09.666Z] [Pipeline] cleanWs
[2026-09-03T03:45:09.719Z] [WS-CLEANUP] Deleting project workspace...
[2026-09-03T03:45:09.719Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[2026-09-03T03:45:09.742Z] [WS-CLEANUP] done
[2026-09-03T03:45:09.772Z] [Pipeline] lock
[2026-09-03T03:45:09.772Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT]
[2026-09-03T03:45:09.772Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created.
[2026-09-03T03:45:09.773Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT]
[2026-09-03T03:45:09.773Z] [Pipeline] {
[2026-09-03T03:45:09.777Z] [Pipeline] }
[2026-09-03T03:45:09.777Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT]
[2026-09-03T03:45:09.780Z] [Pipeline] // lock
[2026-09-03T03:45:09.785Z] [Pipeline] }
[2026-09-03T03:45:09.791Z] [Pipeline] // script
[2026-09-03T03:45:09.793Z] [Pipeline] }
[2026-09-03T03:45:09.795Z] [Pipeline] // stage
[2026-09-03T03:45:09.800Z] [Pipeline] stage
[2026-09-03T03:45:09.801Z] [Pipeline] { (Run functional testing)
[2026-09-03T03:45:09.810Z] [Pipeline] script
[2026-09-03T03:45:09.812Z] [Pipeline] {
[2026-09-03T03:45:09.833Z] [Pipeline] echo
[2026-09-03T03:45:09.833Z] runWithTimeout 360
[2026-09-03T03:45:09.834Z] [Pipeline] timeout
[2026-09-03T03:45:09.834Z] Timeout set to expire in 6 hr 0 min
[2026-09-03T03:45:09.834Z] [Pipeline] {
[2026-09-03T03:45:09.839Z] [Pipeline] sh
[2026-09-03T03:45:10.342Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 ci ci all
[2026-09-03T03:45:10.342Z] CITBRANCH:  main
[2026-09-03T03:45:10.342Z] CITHOME:    /root/ci-tools
[2026-09-03T03:45:10.342Z] CITCMD:     ci-setup-anvil-bm-vm
[2026-09-03T03:45:10.342Z] CITCMDARGS: rhel-9 ci ci all
[2026-09-03T03:45:10.342Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-03T03:45:10.342Z] Genering ssh keys and ssh config
[2026-09-03T03:45:11.075Z] Delay build by 180 seconds to allow Jenkins to cancel the job
[2026-09-03T03:48:20.112Z] Destroying VMs (if any)
[2026-09-03T03:48:20.112Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-03T03:48:20.112Z] Genering ssh keys and ssh config
[2026-09-03T03:48:20.112Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-03T03:48:20.112Z] receiving incremental file list
[2026-09-03T03:48:20.112Z] 
[2026-09-03T03:48:20.112Z] sent 27 bytes  received 509 bytes  1,072.00 bytes/sec
[2026-09-03T03:48:20.112Z] total size is 19,109,445,705  speedup is 35,651,950.94
[2026-09-03T03:48:20.112Z] Generating daily cache
[2026-09-03T03:48:20.112Z] Updating daily cache for rhel-9 2026-09-02
[2026-09-03T03:48:20.112Z] Destroying VMs (if any)
[2026-09-03T03:48:20.112Z] Formatting 'rhel-9-latest.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/images/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:48:20.112Z] wrote an-cache-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:48:20.112Z] 
[2026-09-03T03:48:20.112Z] Starting install...
[2026-09-03T03:48:20.112Z] 
[2026-09-03T03:48:20.112Z] Creating domain...                                          |         00:00     
[2026-09-03T03:48:20.112Z] Domain creation completed.
[2026-09-03T03:48:20.112Z] Waiting for nodes and cloud-init to complete full init
[2026-09-03T03:48:20.112Z] cloud-init status node an-cache: 
[2026-09-03T03:48:20.112Z] Wed Sep  2 23:48:16 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:48:31.091Z] cloud-init status node an-cache: 
[2026-09-03T03:48:31.091Z] Wed Sep  2 23:48:29 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:48:42.170Z] cloud-init status node an-cache: running
[2026-09-03T03:48:42.170Z] Wed Sep  2 23:48:40 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:48:51.320Z] cloud-init status node an-cache: running
[2026-09-03T03:48:51.320Z] Wed Sep  2 23:48:51 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:02.244Z] cloud-init status node an-cache: running
[2026-09-03T03:49:02.244Z] Wed Sep  2 23:49:01 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:12.801Z] cloud-init status node an-cache: running
[2026-09-03T03:49:12.801Z] Wed Sep  2 23:49:12 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:24.681Z] cloud-init status node an-cache: running
[2026-09-03T03:49:24.681Z] Wed Sep  2 23:49:23 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:35.738Z] cloud-init status node an-cache: running
[2026-09-03T03:49:35.738Z] Wed Sep  2 23:49:33 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:45.114Z] cloud-init status node an-cache: running
[2026-09-03T03:49:45.114Z] Wed Sep  2 23:49:44 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:49:56.836Z] cloud-init status node an-cache: running
[2026-09-03T03:49:56.836Z] Wed Sep  2 23:49:55 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:50:06.794Z] cloud-init status node an-cache: running
[2026-09-03T03:50:06.794Z] Wed Sep  2 23:50:06 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:50:19.143Z] cloud-init status node an-cache: running
[2026-09-03T03:50:19.143Z] Wed Sep  2 23:50:17 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:50:28.844Z] cloud-init status node an-cache: running
[2026-09-03T03:50:28.844Z] Wed Sep  2 23:50:27 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:50:39.039Z] cloud-init status node an-cache: running
[2026-09-03T03:50:39.039Z] Wed Sep  2 23:50:38 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:50:49.760Z] cloud-init status node an-cache: running
[2026-09-03T03:50:49.760Z] Wed Sep  2 23:50:49 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:01.729Z] cloud-init status node an-cache: running
[2026-09-03T03:51:01.729Z] Wed Sep  2 23:51:00 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:16.916Z] cloud-init status node an-cache: running
[2026-09-03T03:51:16.916Z] Wed Sep  2 23:51:15 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:26.861Z] cloud-init status node an-cache: running
[2026-09-03T03:51:26.861Z] Wed Sep  2 23:51:26 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:37.914Z] cloud-init status node an-cache: running
[2026-09-03T03:51:37.914Z] Wed Sep  2 23:51:37 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:48.928Z] cloud-init status node an-cache: running
[2026-09-03T03:51:48.928Z] Wed Sep  2 23:51:48 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:51:59.164Z] cloud-init status node an-cache: running
[2026-09-03T03:51:59.164Z] Wed Sep  2 23:51:58 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:52:10.200Z] cloud-init status node an-cache: running
[2026-09-03T03:52:10.200Z] Wed Sep  2 23:52:09 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:52:21.807Z] cloud-init status node an-cache: running
[2026-09-03T03:52:21.807Z] Wed Sep  2 23:52:20 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:52:31.442Z] cloud-init status node an-cache: running
[2026-09-03T03:52:31.442Z] Wed Sep  2 23:52:31 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:52:43.064Z] cloud-init status node an-cache: running
[2026-09-03T03:52:43.064Z] Wed Sep  2 23:52:41 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:52:53.100Z] cloud-init status node an-cache: running
[2026-09-03T03:52:53.100Z] Wed Sep  2 23:52:52 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:03.782Z] cloud-init status node an-cache: running
[2026-09-03T03:53:03.782Z] Wed Sep  2 23:53:03 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:15.060Z] cloud-init status node an-cache: running
[2026-09-03T03:53:15.060Z] Wed Sep  2 23:53:14 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:25.991Z] cloud-init status node an-cache: running
[2026-09-03T03:53:25.991Z] Wed Sep  2 23:53:24 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:37.340Z] cloud-init status node an-cache: running
[2026-09-03T03:53:37.340Z] Wed Sep  2 23:53:35 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:46.992Z] cloud-init status node an-cache: running
[2026-09-03T03:53:46.992Z] Wed Sep  2 23:53:46 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:53:58.832Z] cloud-init status node an-cache: running
[2026-09-03T03:53:58.832Z] Wed Sep  2 23:53:56 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:54:08.210Z] cloud-init status node an-cache: running
[2026-09-03T03:54:08.210Z] Wed Sep  2 23:54:07 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:54:19.758Z] cloud-init status node an-cache: running
[2026-09-03T03:54:19.758Z] Wed Sep  2 23:54:18 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:54:31.100Z] cloud-init status node an-cache: running
[2026-09-03T03:54:31.100Z] Wed Sep  2 23:54:29 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:54:40.136Z] cloud-init status node an-cache: running
[2026-09-03T03:54:40.136Z] Wed Sep  2 23:54:40 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:54:51.125Z] cloud-init status node an-cache: running
[2026-09-03T03:54:51.125Z] Wed Sep  2 23:54:50 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:02.706Z] cloud-init status node an-cache: running
[2026-09-03T03:55:02.706Z] Wed Sep  2 23:55:01 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:13.749Z] cloud-init status node an-cache: running
[2026-09-03T03:55:13.749Z] Wed Sep  2 23:55:12 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:24.729Z] cloud-init status node an-cache: running
[2026-09-03T03:55:24.729Z] Wed Sep  2 23:55:23 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:33.960Z] cloud-init status node an-cache: running
[2026-09-03T03:55:33.960Z] Wed Sep  2 23:55:33 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:44.950Z] cloud-init status node an-cache: running
[2026-09-03T03:55:44.950Z] Wed Sep  2 23:55:44 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:55:55.921Z] cloud-init status node an-cache: running
[2026-09-03T03:55:55.921Z] Wed Sep  2 23:55:55 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:56:07.050Z] cloud-init status node an-cache: running
[2026-09-03T03:56:07.050Z] Wed Sep  2 23:56:06 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:56:18.031Z] cloud-init status node an-cache: running
[2026-09-03T03:56:18.031Z] Wed Sep  2 23:56:16 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:56:29.007Z] cloud-init status node an-cache: running
[2026-09-03T03:56:29.007Z] Wed Sep  2 23:56:27 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:56:40.122Z] cloud-init status node an-cache: running
[2026-09-03T03:56:40.122Z] Wed Sep  2 23:56:38 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:56:49.157Z] cloud-init status node an-cache: running
[2026-09-03T03:56:49.498Z] Wed Sep  2 23:56:49 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:00.475Z] cloud-init status node an-cache: running
[2026-09-03T03:57:00.475Z] Wed Sep  2 23:56:59 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:11.452Z] cloud-init status node an-cache: running
[2026-09-03T03:57:11.452Z] Wed Sep  2 23:57:10 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:22.424Z] cloud-init status node an-cache: running
[2026-09-03T03:57:22.424Z] Wed Sep  2 23:57:21 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:33.398Z] cloud-init status node an-cache: running
[2026-09-03T03:57:33.398Z] Wed Sep  2 23:57:32 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:44.507Z] cloud-init status node an-cache: running
[2026-09-03T03:57:44.507Z] Wed Sep  2 23:57:43 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:57:55.493Z] cloud-init status node an-cache: running
[2026-09-03T03:57:55.493Z] Wed Sep  2 23:57:53 EDT 2026 actual: 0 expected: 1
[2026-09-03T03:58:04.524Z] cloud-init status node an-cache: done
[2026-09-03T03:58:04.524Z] Wed Sep  2 23:58:04 EDT 2026 actual: 1 expected: 1
[2026-09-03T03:59:02.629Z] Disconnecting an-cache.ci.alteeve.com from Red Hat.
[2026-09-03T03:59:02.629Z] This might take a few seconds.
[2026-09-03T03:59:02.629Z] 
[2026-09-03T03:59:02.629Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T03:59:02.629Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T03:59:02.629Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T03:59:02.629Z] 
[2026-09-03T03:59:02.629Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T03:59:02.972Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T03:59:02.972Z] 
[2026-09-03T03:59:02.972Z] 1. Register with subscription-manager
[2026-09-03T03:59:02.972Z] # subscription-manager register
[2026-09-03T03:59:02.972Z] 
[2026-09-03T03:59:02.972Z] 2. Register with insights-client
[2026-09-03T03:59:02.972Z] # insights-client --register
[2026-09-03T03:59:03.706Z] This system is currently not registered.
[2026-09-03T03:59:03.706Z] Domain 'an-cache' is being shutdown
[2026-09-03T03:59:03.706Z] 
[2026-09-03T03:59:09.732Z] Domain 'an-cache' has been undefined
[2026-09-03T03:59:09.732Z] 
[2026-09-03T03:59:09.732Z] Done updating daily cache
[2026-09-03T03:59:09.732Z] Creating nodes disks
[2026-09-03T03:59:14.703Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-03T03:59:14.703Z] Generating an-test-deployX kickstart files/isos
[2026-09-03T03:59:14.703Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-03T03:59:14.703Z] 
[2026-09-03T03:59:14.703Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso'
[2026-09-03T03:59:14.703Z] Media current: stdio file, overwriteable
[2026-09-03T03:59:14.703Z] Media status : is blank
[2026-09-03T03:59:14.703Z] Media summary: 0 sessions, 0 data blocks, 0 data,  802g free
[2026-09-03T03:59:14.703Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1'
[2026-09-03T03:59:14.703Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:14.703Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] ISO image produced: 184 sectors
[2026-09-03T03:59:15.045Z] Written to medium : 184 sectors at LBA 0
[2026-09-03T03:59:15.045Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso'
[2026-09-03T03:59:15.045Z] Media current: stdio file, overwriteable
[2026-09-03T03:59:15.045Z] Media status : is blank
[2026-09-03T03:59:15.045Z] Media summary: 0 sessions, 0 data blocks, 0 data,  802g free
[2026-09-03T03:59:15.045Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2'
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] ISO image produced: 184 sectors
[2026-09-03T03:59:15.045Z] Written to medium : 184 sectors at LBA 0
[2026-09-03T03:59:15.045Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso'
[2026-09-03T03:59:15.045Z] Media current: stdio file, overwriteable
[2026-09-03T03:59:15.045Z] Media status : is blank
[2026-09-03T03:59:15.045Z] Media summary: 0 sessions, 0 data blocks, 0 data,  802g free
[2026-09-03T03:59:15.045Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3'
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] ISO image produced: 184 sectors
[2026-09-03T03:59:15.045Z] Written to medium : 184 sectors at LBA 0
[2026-09-03T03:59:15.045Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-03T03:59:15.045Z] 
[2026-09-03T03:59:15.045Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso'
[2026-09-03T03:59:15.045Z] Media current: stdio file, overwriteable
[2026-09-03T03:59:15.045Z] Media status : is blank
[2026-09-03T03:59:15.045Z] Media summary: 0 sessions, 0 data blocks, 0 data,  802g free
[2026-09-03T03:59:15.045Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4'
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.045Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.387Z] ISO image produced: 184 sectors
[2026-09-03T03:59:15.387Z] Written to medium : 184 sectors at LBA 0
[2026-09-03T03:59:15.387Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully.
[2026-09-03T03:59:15.387Z] 
[2026-09-03T03:59:15.387Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-03T03:59:15.387Z] 
[2026-09-03T03:59:15.387Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso'
[2026-09-03T03:59:15.387Z] Media current: stdio file, overwriteable
[2026-09-03T03:59:15.387Z] Media status : is blank
[2026-09-03T03:59:15.387Z] Media summary: 0 sessions, 0 data blocks, 0 data,  802g free
[2026-09-03T03:59:15.387Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5'
[2026-09-03T03:59:15.387Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.387Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-03T03:59:15.387Z] ISO image produced: 184 sectors
[2026-09-03T03:59:15.387Z] Written to medium : 184 sectors at LBA 0
[2026-09-03T03:59:15.387Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully.
[2026-09-03T03:59:15.387Z] 
[2026-09-03T03:59:15.387Z] Generating cloud-init configs
[2026-09-03T03:59:15.387Z] Generating cloud-init metadata and seed images
[2026-09-03T03:59:15.387Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:59:15.730Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:59:15.730Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:59:16.072Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:59:16.072Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-03T03:59:16.072Z] Creating VMs
[2026-09-03T03:59:17.259Z] 
[2026-09-03T03:59:17.259Z] Starting install...
[2026-09-03T03:59:17.259Z] 
[2026-09-03T03:59:17.259Z] Creating domain...                                          |         00:00     
[2026-09-03T03:59:17.259Z] Domain creation completed.
[2026-09-03T03:59:18.447Z] 
[2026-09-03T03:59:18.447Z] Starting install...
[2026-09-03T03:59:18.447Z] 
[2026-09-03T03:59:18.447Z] Creating domain...                                          |         00:00     
[2026-09-03T03:59:18.447Z] Domain creation completed.
[2026-09-03T03:59:20.158Z] 
[2026-09-03T03:59:20.158Z] Starting install...
[2026-09-03T03:59:20.158Z] 
[2026-09-03T03:59:20.158Z] Creating domain...                                          |         00:00     
[2026-09-03T03:59:20.158Z] Domain creation completed.
[2026-09-03T03:59:21.345Z] 
[2026-09-03T03:59:21.345Z] Starting install...
[2026-09-03T03:59:21.345Z] 
[2026-09-03T03:59:21.345Z] Creating domain...                                          |         00:00     
[2026-09-03T03:59:21.345Z] Domain creation completed.
[2026-09-03T03:59:23.054Z] 
[2026-09-03T03:59:23.054Z] Starting install...
[2026-09-03T03:59:23.054Z] 
[2026-09-03T03:59:23.054Z] Creating domain...                                          |         00:00     
[2026-09-03T03:59:23.054Z] Domain creation completed.
[2026-09-03T03:59:23.054Z] Waiting for nodes and cloud-init to complete full init
[2026-09-03T03:59:26.086Z] cloud-init status node an-striker01: 
[2026-09-03T03:59:29.120Z] cloud-init status node an-striker02: 
[2026-09-03T03:59:32.265Z] cloud-init status node an-a01n01: 
[2026-09-03T03:59:35.300Z] cloud-init status node an-a01n02: 
[2026-09-03T03:59:38.331Z] cloud-init status node an-a01dr01: 
[2026-09-03T03:59:38.331Z] Wed Sep  2 23:59:37 EDT 2026 actual: 0 expected: 5
[2026-09-03T03:59:49.314Z] cloud-init status node an-striker01: running
[2026-09-03T03:59:50.501Z] cloud-init status node an-striker02: running
[2026-09-03T03:59:51.801Z] cloud-init status node an-a01n01: running
[2026-09-03T03:59:52.536Z] cloud-init status node an-a01n02: running
[2026-09-03T03:59:53.722Z] cloud-init status node an-a01dr01: running
[2026-09-03T03:59:53.722Z] Wed Sep  2 23:59:53 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:00:04.705Z] cloud-init status node an-striker01: running
[2026-09-03T04:00:05.438Z] cloud-init status node an-striker02: running
[2026-09-03T04:00:06.174Z] cloud-init status node an-a01n01: running
[2026-09-03T04:00:06.908Z] cloud-init status node an-a01n02: running
[2026-09-03T04:00:07.643Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:00:07.643Z] Thu Sep  3 00:00:07 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:00:18.773Z] cloud-init status node an-striker01: running
[2026-09-03T04:00:19.507Z] cloud-init status node an-striker02: running
[2026-09-03T04:00:20.242Z] cloud-init status node an-a01n01: running
[2026-09-03T04:00:20.976Z] cloud-init status node an-a01n02: running
[2026-09-03T04:00:22.162Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:00:22.162Z] Thu Sep  3 00:00:21 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:00:33.142Z] cloud-init status node an-striker01: running
[2026-09-03T04:00:33.484Z] cloud-init status node an-striker02: running
[2026-09-03T04:00:34.672Z] cloud-init status node an-a01n01: running
[2026-09-03T04:00:35.407Z] cloud-init status node an-a01n02: running
[2026-09-03T04:00:36.593Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:00:36.593Z] Thu Sep  3 00:00:36 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:00:47.692Z] cloud-init status node an-striker01: running
[2026-09-03T04:00:48.035Z] cloud-init status node an-striker02: running
[2026-09-03T04:00:49.223Z] cloud-init status node an-a01n01: running
[2026-09-03T04:00:49.959Z] cloud-init status node an-a01n02: running
[2026-09-03T04:00:50.692Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:00:50.692Z] Thu Sep  3 00:00:50 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:01:01.681Z] cloud-init status node an-striker01: running
[2026-09-03T04:01:02.418Z] cloud-init status node an-striker02: running
[2026-09-03T04:01:03.621Z] cloud-init status node an-a01n01: running
[2026-09-03T04:01:04.369Z] cloud-init status node an-a01n02: running
[2026-09-03T04:01:05.108Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:01:05.108Z] Thu Sep  3 00:01:05 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:01:16.106Z] cloud-init status node an-striker01: running
[2026-09-03T04:01:17.294Z] cloud-init status node an-striker02: running
[2026-09-03T04:01:18.028Z] cloud-init status node an-a01n01: running
[2026-09-03T04:01:19.250Z] cloud-init status node an-a01n02: running
[2026-09-03T04:01:19.983Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:01:19.983Z] Thu Sep  3 00:01:19 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:01:31.403Z] cloud-init status node an-striker01: running
[2026-09-03T04:01:32.145Z] cloud-init status node an-striker02: running
[2026-09-03T04:01:32.888Z] cloud-init status node an-a01n01: running
[2026-09-03T04:01:34.107Z] cloud-init status node an-a01n02: running
[2026-09-03T04:01:40.681Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:01:40.681Z] Thu Sep  3 00:01:39 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:01:51.790Z] cloud-init status node an-striker01: running
[2026-09-03T04:01:52.144Z] cloud-init status node an-striker02: running
[2026-09-03T04:01:52.883Z] cloud-init status node an-a01n01: running
[2026-09-03T04:01:54.201Z] cloud-init status node an-a01n02: running
[2026-09-03T04:01:54.948Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:01:54.948Z] Thu Sep  3 00:01:54 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:02:05.923Z] cloud-init status node an-striker01: running
[2026-09-03T04:02:06.657Z] cloud-init status node an-striker02: running
[2026-09-03T04:02:07.843Z] cloud-init status node an-a01n01: running
[2026-09-03T04:02:08.578Z] cloud-init status node an-a01n02: running
[2026-09-03T04:02:09.312Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:02:09.312Z] Thu Sep  3 00:02:09 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:02:20.299Z] cloud-init status node an-striker01: running
[2026-09-03T04:02:21.145Z] cloud-init status node an-striker02: running
[2026-09-03T04:02:21.880Z] cloud-init status node an-a01n01: running
[2026-09-03T04:02:23.069Z] cloud-init status node an-a01n02: running
[2026-09-03T04:02:23.804Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:02:23.804Z] Thu Sep  3 00:02:23 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:02:34.791Z] cloud-init status node an-striker01: running
[2026-09-03T04:02:35.527Z] cloud-init status node an-striker02: running
[2026-09-03T04:02:36.712Z] cloud-init status node an-a01n01: running
[2026-09-03T04:02:37.448Z] cloud-init status node an-a01n02: running
[2026-09-03T04:02:38.184Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:02:38.184Z] Thu Sep  3 00:02:38 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:02:49.277Z] cloud-init status node an-striker01: running
[2026-09-03T04:02:50.013Z] cloud-init status node an-striker02: running
[2026-09-03T04:02:51.202Z] cloud-init status node an-a01n01: running
[2026-09-03T04:02:52.388Z] cloud-init status node an-a01n02: running
[2026-09-03T04:02:53.575Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:02:53.575Z] Thu Sep  3 00:02:53 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:03:04.558Z] cloud-init status node an-striker01: running
[2026-09-03T04:03:05.291Z] cloud-init status node an-striker02: running
[2026-09-03T04:03:06.028Z] cloud-init status node an-a01n01: running
[2026-09-03T04:03:06.859Z] cloud-init status node an-a01n02: running
[2026-09-03T04:03:08.046Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:03:08.046Z] Thu Sep  3 00:03:07 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:03:19.027Z] cloud-init status node an-striker01: running
[2026-09-03T04:03:19.765Z] cloud-init status node an-striker02: running
[2026-09-03T04:03:20.954Z] cloud-init status node an-a01n01: running
[2026-09-03T04:03:21.690Z] cloud-init status node an-a01n02: running
[2026-09-03T04:03:22.882Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:03:22.882Z] Thu Sep  3 00:03:22 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:03:33.958Z] cloud-init status node an-striker01: running
[2026-09-03T04:03:34.694Z] cloud-init status node an-striker02: running
[2026-09-03T04:03:35.883Z] cloud-init status node an-a01n01: running
[2026-09-03T04:03:36.617Z] cloud-init status node an-a01n02: running
[2026-09-03T04:03:37.804Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:03:37.804Z] Thu Sep  3 00:03:37 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:03:48.786Z] cloud-init status node an-striker01: running
[2026-09-03T04:03:49.521Z] cloud-init status node an-striker02: running
[2026-09-03T04:03:50.709Z] cloud-init status node an-a01n01: running
[2026-09-03T04:03:51.898Z] cloud-init status node an-a01n02: running
[2026-09-03T04:03:52.633Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:03:52.633Z] Thu Sep  3 00:03:52 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:04:03.613Z] cloud-init status node an-striker01: running
[2026-09-03T04:04:04.352Z] cloud-init status node an-striker02: running
[2026-09-03T04:04:05.540Z] cloud-init status node an-a01n01: running
[2026-09-03T04:04:06.277Z] cloud-init status node an-a01n02: running
[2026-09-03T04:04:07.469Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:04:07.469Z] Thu Sep  3 00:04:07 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:04:18.455Z] cloud-init status node an-striker01: running
[2026-09-03T04:04:19.282Z] cloud-init status node an-striker02: running
[2026-09-03T04:04:20.018Z] cloud-init status node an-a01n01: running
[2026-09-03T04:04:21.213Z] cloud-init status node an-a01n02: running
[2026-09-03T04:04:22.406Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:04:22.406Z] Thu Sep  3 00:04:21 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:04:33.388Z] cloud-init status node an-striker01: running
[2026-09-03T04:04:34.124Z] cloud-init status node an-striker02: running
[2026-09-03T04:04:35.311Z] cloud-init status node an-a01n01: running
[2026-09-03T04:04:36.048Z] cloud-init status node an-a01n02: running
[2026-09-03T04:04:37.236Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:04:37.236Z] Thu Sep  3 00:04:36 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:04:48.314Z] cloud-init status node an-striker01: running
[2026-09-03T04:04:49.048Z] cloud-init status node an-striker02: running
[2026-09-03T04:04:50.235Z] cloud-init status node an-a01n01: running
[2026-09-03T04:04:50.970Z] cloud-init status node an-a01n02: running
[2026-09-03T04:04:52.159Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:04:52.159Z] Thu Sep  3 00:04:51 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:05:03.144Z] cloud-init status node an-striker01: running
[2026-09-03T04:05:03.881Z] cloud-init status node an-striker02: running
[2026-09-03T04:05:04.616Z] cloud-init status node an-a01n01: running
[2026-09-03T04:05:05.808Z] cloud-init status node an-a01n02: running
[2026-09-03T04:05:06.997Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:05:06.997Z] Thu Sep  3 00:05:06 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:05:18.005Z] cloud-init status node an-striker01: running
[2026-09-03T04:05:18.740Z] cloud-init status node an-striker02: running
[2026-09-03T04:05:19.474Z] cloud-init status node an-a01n01: running
[2026-09-03T04:05:20.662Z] cloud-init status node an-a01n02: running
[2026-09-03T04:05:21.398Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:05:21.398Z] Thu Sep  3 00:05:21 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:05:32.481Z] cloud-init status node an-striker01: running
[2026-09-03T04:05:33.667Z] cloud-init status node an-striker02: running
[2026-09-03T04:05:34.401Z] cloud-init status node an-a01n01: running
[2026-09-03T04:05:35.135Z] cloud-init status node an-a01n02: running
[2026-09-03T04:05:36.320Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:05:36.320Z] Thu Sep  3 00:05:35 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:05:47.300Z] cloud-init status node an-striker01: running
[2026-09-03T04:05:48.037Z] cloud-init status node an-striker02: running
[2026-09-03T04:05:49.227Z] cloud-init status node an-a01n01: running
[2026-09-03T04:05:49.962Z] cloud-init status node an-a01n02: running
[2026-09-03T04:05:50.697Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:05:50.697Z] Thu Sep  3 00:05:50 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:06:06.839Z] cloud-init status node an-striker01: running
[2026-09-03T04:06:07.182Z] cloud-init status node an-striker02: running
[2026-09-03T04:06:08.371Z] cloud-init status node an-a01n01: running
[2026-09-03T04:06:09.560Z] cloud-init status node an-a01n02: running
[2026-09-03T04:06:10.749Z] cloud-init status node an-a01dr01: running
[2026-09-03T04:06:10.749Z] Thu Sep  3 00:06:10 EDT 2026 actual: 0 expected: 5
[2026-09-03T04:06:24.059Z] cloud-init status node an-striker01: running
[2026-09-03T04:06:24.059Z] cloud-init status node an-striker02: running
[2026-09-03T04:06:24.804Z] cloud-init status node an-a01n01: done
[2026-09-03T04:06:25.545Z] cloud-init status node an-a01n02: done
[2026-09-03T04:06:26.284Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:06:26.284Z] Thu Sep  3 00:06:26 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:06:37.299Z] cloud-init status node an-striker01: running
[2026-09-03T04:06:38.490Z] cloud-init status node an-striker02: running
[2026-09-03T04:06:39.684Z] cloud-init status node an-a01n01: done
[2026-09-03T04:06:40.422Z] cloud-init status node an-a01n02: done
[2026-09-03T04:06:41.615Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:06:41.615Z] Thu Sep  3 00:06:41 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:06:52.615Z] cloud-init status node an-striker01: running
[2026-09-03T04:06:53.806Z] cloud-init status node an-striker02: running
[2026-09-03T04:06:54.544Z] cloud-init status node an-a01n01: done
[2026-09-03T04:06:55.733Z] cloud-init status node an-a01n02: done
[2026-09-03T04:06:56.469Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:06:56.469Z] Thu Sep  3 00:06:56 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:07:09.766Z] cloud-init status node an-striker01: running
[2026-09-03T04:07:09.766Z] cloud-init status node an-striker02: running
[2026-09-03T04:07:09.766Z] cloud-init status node an-a01n01: done
[2026-09-03T04:07:10.957Z] cloud-init status node an-a01n02: done
[2026-09-03T04:07:11.694Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:07:11.694Z] Thu Sep  3 00:07:11 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:07:22.778Z] cloud-init status node an-striker01: running
[2026-09-03T04:07:23.968Z] cloud-init status node an-striker02: running
[2026-09-03T04:07:24.704Z] cloud-init status node an-a01n01: done
[2026-09-03T04:07:25.897Z] cloud-init status node an-a01n02: done
[2026-09-03T04:07:26.633Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:07:26.633Z] Thu Sep  3 00:07:26 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:07:37.629Z] cloud-init status node an-striker01: running
[2026-09-03T04:07:38.817Z] cloud-init status node an-striker02: running
[2026-09-03T04:07:39.648Z] cloud-init status node an-a01n01: done
[2026-09-03T04:07:40.383Z] cloud-init status node an-a01n02: done
[2026-09-03T04:07:41.573Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:07:41.573Z] Thu Sep  3 00:07:41 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:07:52.576Z] cloud-init status node an-striker01: running
[2026-09-03T04:07:53.765Z] cloud-init status node an-striker02: running
[2026-09-03T04:07:54.501Z] cloud-init status node an-a01n01: done
[2026-09-03T04:07:55.687Z] cloud-init status node an-a01n02: done
[2026-09-03T04:07:56.423Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:07:56.423Z] Thu Sep  3 00:07:56 EDT 2026 actual: 3 expected: 5
[2026-09-03T04:08:09.833Z] cloud-init status node an-striker01: done
[2026-09-03T04:08:09.833Z] cloud-init status node an-striker02: done
[2026-09-03T04:08:10.175Z] cloud-init status node an-a01n01: done
[2026-09-03T04:08:11.362Z] cloud-init status node an-a01n02: done
[2026-09-03T04:08:12.095Z] cloud-init status node an-a01dr01: done
[2026-09-03T04:08:12.095Z] Thu Sep  3 00:08:12 EDT 2026 actual: 5 expected: 5
[2026-09-03T04:08:12.095Z] Building simengine model
[2026-09-03T04:08:25.386Z] Generating anvil striker auto config
[2026-09-03T04:08:28.425Z] Cloning into 'anvil-test-suite'...
[2026-09-03T04:08:29.160Z] generated ansible config:
[2026-09-03T04:08:29.160Z] =========================
[2026-09-03T04:08:29.160Z] [defaults]
[2026-09-03T04:08:29.160Z] inventory = ansible-inventory
[2026-09-03T04:08:29.160Z] forks = 500
[2026-09-03T04:08:29.160Z] any_errors_fatal = true
[2026-09-03T04:08:29.160Z] display_args_to_stdout = true
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:29.160Z] [ssh_connection]
[2026-09-03T04:08:29.160Z] retries = 20
[2026-09-03T04:08:29.160Z] =========================
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:29.160Z] generated ansible inventory:
[2026-09-03T04:08:29.160Z] ============================
[2026-09-03T04:08:29.160Z] [strikers]
[2026-09-03T04:08:29.160Z] an-striker01
[2026-09-03T04:08:29.160Z] an-striker02
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:29.160Z] [nodes]
[2026-09-03T04:08:29.160Z] an-a01n01
[2026-09-03T04:08:29.160Z] an-a01n02
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:29.160Z] [drnodes]
[2026-09-03T04:08:29.160Z] an-a01dr01
[2026-09-03T04:08:29.160Z] ============================
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:29.160Z] Checking connectivity to nodes
[2026-09-03T04:08:29.160Z] 
[2026-09-03T04:08:31.575Z] an-a01dr01 | SUCCESS => {
[2026-09-03T04:08:31.575Z]     "ansible_facts": {
[2026-09-03T04:08:31.575Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-03T04:08:31.575Z]     },
[2026-09-03T04:08:31.575Z]     "changed": false,
[2026-09-03T04:08:31.575Z]     "ping": "pong"
[2026-09-03T04:08:31.575Z] }
[2026-09-03T04:08:31.575Z] an-a01n02 | SUCCESS => {
[2026-09-03T04:08:31.575Z]     "ansible_facts": {
[2026-09-03T04:08:31.575Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-03T04:08:31.575Z]     },
[2026-09-03T04:08:31.575Z]     "changed": false,
[2026-09-03T04:08:31.575Z]     "ping": "pong"
[2026-09-03T04:08:31.575Z] }
[2026-09-03T04:08:31.575Z] an-a01n01 | SUCCESS => {
[2026-09-03T04:08:31.575Z]     "ansible_facts": {
[2026-09-03T04:08:31.575Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-03T04:08:31.575Z]     },
[2026-09-03T04:08:31.575Z]     "changed": false,
[2026-09-03T04:08:31.575Z]     "ping": "pong"
[2026-09-03T04:08:31.575Z] }
[2026-09-03T04:08:31.575Z] an-striker01 | SUCCESS => {
[2026-09-03T04:08:31.575Z]     "ansible_facts": {
[2026-09-03T04:08:31.575Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-03T04:08:31.575Z]     },
[2026-09-03T04:08:31.575Z]     "changed": false,
[2026-09-03T04:08:31.575Z]     "ping": "pong"
[2026-09-03T04:08:31.575Z] }
[2026-09-03T04:08:31.575Z] an-striker02 | SUCCESS => {
[2026-09-03T04:08:31.575Z]     "ansible_facts": {
[2026-09-03T04:08:31.575Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-03T04:08:31.575Z]     },
[2026-09-03T04:08:31.575Z]     "changed": false,
[2026-09-03T04:08:31.575Z]     "ping": "pong"
[2026-09-03T04:08:31.575Z] }
[2026-09-03T04:08:31.575Z] 
[2026-09-03T04:08:32.313Z] 
[2026-09-03T04:08:32.313Z] PLAY [Setup all nodes] *********************************************************
[2026-09-03T04:08:32.313Z] 
[2026-09-03T04:08:32.313Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:08:34.636Z] ok: [an-a01n01]
[2026-09-03T04:08:34.636Z] ok: [an-a01n02]
[2026-09-03T04:08:34.636Z] ok: [an-a01dr01]
[2026-09-03T04:08:34.636Z] ok: [an-striker01]
[2026-09-03T04:08:34.636Z] ok: [an-striker02]
[2026-09-03T04:08:34.636Z] 
[2026-09-03T04:08:34.636Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] ***
[2026-09-03T04:08:35.823Z] changed: [an-a01n01]
[2026-09-03T04:08:35.823Z] changed: [an-a01n02]
[2026-09-03T04:08:35.823Z] changed: [an-a01dr01]
[2026-09-03T04:08:35.823Z] changed: [an-striker02]
[2026-09-03T04:08:35.823Z] changed: [an-striker01]
[2026-09-03T04:08:35.823Z] 
[2026-09-03T04:08:35.823Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] ***
[2026-09-03T04:08:36.564Z] ok: [an-a01n01]
[2026-09-03T04:08:36.564Z] ok: [an-a01n02]
[2026-09-03T04:08:36.564Z] ok: [an-a01dr01]
[2026-09-03T04:08:36.564Z] ok: [an-striker01]
[2026-09-03T04:08:36.564Z] ok: [an-striker02]
[2026-09-03T04:08:36.564Z] 
[2026-09-03T04:08:36.564Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] ***
[2026-09-03T04:08:38.276Z] changed: [an-a01n01]
[2026-09-03T04:08:38.276Z] changed: [an-a01dr01]
[2026-09-03T04:08:38.276Z] changed: [an-striker02]
[2026-09-03T04:08:38.276Z] changed: [an-striker01]
[2026-09-03T04:08:38.276Z] changed: [an-a01n02]
[2026-09-03T04:08:38.276Z] 
[2026-09-03T04:08:38.276Z] PLAY [setup strikers] **********************************************************
[2026-09-03T04:08:38.276Z] 
[2026-09-03T04:08:38.276Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:08:39.988Z] ok: [an-striker02]
[2026-09-03T04:08:39.988Z] ok: [an-striker01]
[2026-09-03T04:08:39.988Z] 
[2026-09-03T04:08:39.988Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] ***
[2026-09-03T04:08:41.700Z] changed: [an-striker01]
[2026-09-03T04:08:41.700Z] changed: [an-striker02]
[2026-09-03T04:08:41.700Z] 
[2026-09-03T04:08:41.700Z] PLAY [setup lvm storage] *******************************************************
[2026-09-03T04:08:41.700Z] 
[2026-09-03T04:08:41.700Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:08:44.021Z] ok: [an-a01n01]
[2026-09-03T04:08:44.021Z] ok: [an-a01dr01]
[2026-09-03T04:08:44.021Z] ok: [an-a01n02]
[2026-09-03T04:08:44.021Z] 
[2026-09-03T04:08:44.021Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg
[2026-09-03T04:08:44.021Z] ] ****************
[2026-09-03T04:08:44.755Z] ok: [an-a01n01]
[2026-09-03T04:08:44.755Z] ok: [an-a01n02]
[2026-09-03T04:08:44.755Z] ok: [an-a01dr01]
[2026-09-03T04:08:44.755Z] 
[2026-09-03T04:08:44.755Z] TASK [vgcheck _raw_params=end_play] ********************************************
[2026-09-03T04:08:44.755Z] skipping: [an-a01n01]
[2026-09-03T04:08:44.755Z] 
[2026-09-03T04:08:44.755Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] *********************
[2026-09-03T04:08:46.032Z] changed: [an-a01n01]
[2026-09-03T04:08:46.032Z] changed: [an-a01n02]
[2026-09-03T04:08:46.032Z] changed: [an-a01dr01]
[2026-09-03T04:08:46.032Z] 
[2026-09-03T04:08:46.032Z] PLAY [deploy anvil] ************************************************************
[2026-09-03T04:08:46.032Z] 
[2026-09-03T04:08:46.032Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:08:47.743Z] ok: [an-striker02]
[2026-09-03T04:08:47.743Z] ok: [an-striker01]
[2026-09-03T04:08:47.743Z] 
[2026-09-03T04:08:47.743Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] ***
[2026-09-03T04:08:48.478Z] ok: [an-striker02]
[2026-09-03T04:08:48.478Z] ok: [an-striker01]
[2026-09-03T04:08:48.478Z] 
[2026-09-03T04:08:48.478Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] ***
[2026-09-03T04:08:49.666Z] changed: [an-striker02]
[2026-09-03T04:08:49.666Z] changed: [an-striker01]
[2026-09-03T04:08:49.666Z] 
[2026-09-03T04:08:49.666Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg
[2026-09-03T04:08:49.666Z] ] ***
[2026-09-03T04:08:52.701Z] changed: [an-striker02]
[2026-09-03T04:08:52.701Z] changed: [an-striker01]
[2026-09-03T04:08:52.701Z] 
[2026-09-03T04:08:52.701Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:08:52.701Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:08:52.701Z]      --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';"
[2026-09-03T04:08:52.701Z] ] ***
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left).
[2026-09-03T04:15:02.941Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left).
[2026-09-03T04:15:02.941Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:15:02.941Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:15:02.941Z] 
[2026-09-03T04:15:02.941Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)"
[2026-09-03T04:15:02.941Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g')
[2026-09-03T04:15:02.941Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l)
[2026-09-03T04:15:02.941Z] if [ "$dbcon" = 1 ]  &&  [ "$dbhosts" = 2 ]; then
[2026-09-03T04:15:02.941Z]   echo "DB has completed sync"
[2026-09-03T04:15:02.941Z]   exit 0
[2026-09-03T04:15:02.941Z] fi
[2026-09-03T04:15:02.941Z] if [ "$dbcon" != 2 ]; then
[2026-09-03T04:15:02.941Z]   echo "Detected $dbcon db connections (expected 2)"
[2026-09-03T04:15:02.941Z]   exit 1
[2026-09-03T04:15:02.941Z] fi
[2026-09-03T04:15:02.941Z] ] ***
[2026-09-03T04:15:36.559Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left).
[2026-09-03T04:15:36.559Z] ok: [an-striker01]
[2026-09-03T04:15:36.559Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left).
[2026-09-03T04:15:36.559Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left).
[2026-09-03T04:15:36.559Z] ok: [an-striker02]
[2026-09-03T04:15:36.559Z] 
[2026-09-03T04:15:36.559Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:15:36.559Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:15:36.559Z]      --username=postgres anvil -t \
[2026-09-03T04:15:36.559Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \
[2026-09-03T04:15:36.559Z] grep altee | \
[2026-09-03T04:15:36.559Z] grep 1$ | \
[2026-09-03T04:15:36.559Z] wc -l
[2026-09-03T04:15:36.559Z] ] ***
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left).
[2026-09-03T04:21:17.155Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left).
[2026-09-03T04:21:17.156Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left).
[2026-09-03T04:25:26.879Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left).
[2026-09-03T04:25:26.879Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:25:26.879Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:25:26.879Z] 
[2026-09-03T04:25:26.879Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:25:26.879Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:25:26.879Z]      --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \
[2026-09-03T04:25:26.879Z] grep altee | \
[2026-09-03T04:25:26.879Z] wc -l
[2026-09-03T04:25:26.879Z] ] ***
[2026-09-03T04:25:26.879Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:25:26.879Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:25:26.879Z] 
[2026-09-03T04:25:26.879Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:25:26.879Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:25:26.879Z]      --username=postgres anvil -t \
[2026-09-03T04:25:26.879Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \
[2026-09-03T04:25:26.879Z] grep altee | \
[2026-09-03T04:25:26.880Z] grep 1$ | \
[2026-09-03T04:25:26.880Z] wc -l
[2026-09-03T04:25:26.880Z] ] ***
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left).
[2026-09-03T04:27:35.569Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left).
[2026-09-03T04:27:35.569Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:27:35.569Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:27:35.569Z] 
[2026-09-03T04:27:35.569Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:27:35.569Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:27:35.569Z]      --username=postgres anvil -t \
[2026-09-03T04:27:35.569Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \
[2026-09-03T04:27:35.569Z] grep altee | \
[2026-09-03T04:27:35.569Z] grep 1$ | \
[2026-09-03T04:27:35.569Z] wc -l
[2026-09-03T04:27:35.569Z] ] ***
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left).
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left).
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left).
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left).
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left).
[2026-09-03T04:27:58.852Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left).
[2026-09-03T04:27:58.852Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:27:58.852Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:27:58.852Z] 
[2026-09-03T04:27:58.852Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-03T04:27:58.852Z] psql --host="{{ inventory_hostname }}" \
[2026-09-03T04:27:58.852Z]      --username=postgres anvil -t \
[2026-09-03T04:27:58.852Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \
[2026-09-03T04:27:58.852Z] grep altee | \
[2026-09-03T04:27:58.852Z] grep 1$ | \
[2026-09-03T04:27:58.852Z] wc -l
[2026-09-03T04:27:58.852Z] ] ***
[2026-09-03T04:27:58.852Z] ok: [an-striker01 -> localhost]
[2026-09-03T04:27:58.852Z] ok: [an-striker02 -> localhost]
[2026-09-03T04:27:58.852Z] 
[2026-09-03T04:27:58.852Z] PLAY [check network status after deployment] ***********************************
[2026-09-03T04:27:58.852Z] 
[2026-09-03T04:27:58.852Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:28:00.564Z] ok: [an-a01n01]
[2026-09-03T04:28:00.564Z] ok: [an-a01n02]
[2026-09-03T04:28:00.564Z] ok: [an-striker01]
[2026-09-03T04:28:00.564Z] ok: [an-striker02]
[2026-09-03T04:28:00.564Z] ok: [an-a01dr01]
[2026-09-03T04:28:00.564Z] 
[2026-09-03T04:28:00.564Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')"
[2026-09-03T04:28:00.564Z] if [ "$device" = "" ]; then
[2026-09-03T04:28:00.564Z]   echo "ERROR: unable to determine device for iface {{ item }}"
[2026-09-03T04:28:00.564Z]   exit 1
[2026-09-03T04:28:00.564Z] fi
[2026-09-03T04:28:00.564Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then
[2026-09-03T04:28:00.564Z]   echo "ERROR: {{ item }} / $device is not set to autoconnecct"
[2026-09-03T04:28:00.564Z]   exit 1
[2026-09-03T04:28:00.564Z] fi
[2026-09-03T04:28:00.564Z] exit 0
[2026-09-03T04:28:00.564Z] ] ***
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=ifn1_bridge1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=sn1_link2)
[2026-09-03T04:28:09.784Z] skipping: [an-a01n02] => (item=lo) 
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=ifn1_bridge1)
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=bcn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=ifn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=sn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=ifn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=bcn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=bcn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=bcn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=ifn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=bcn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=ifn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=ifn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=sn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=sn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=bcn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=ifn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=bcn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=bcn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=ifn1_link1)
[2026-09-03T04:28:09.784Z] skipping: [an-striker01] => (item=lo) 
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=sn1_bond1)
[2026-09-03T04:28:09.784Z] skipping: [an-a01n01] => (item=lo) 
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=bcn1_bridge1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=bcn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=bcn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=ifn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=bcn1_bridge1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=bcn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=sn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=ifn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=ifn1_link2)
[2026-09-03T04:28:09.784Z] skipping: [an-striker02] => (item=lo) 
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=bcn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=sn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=sn1_bond1)
[2026-09-03T04:28:09.784Z] skipping: [an-a01dr01] => (item=lo) 
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=sn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=ifn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-striker02] => (item=bcn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-striker01] => (item=ifn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=ifn1_link2)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=ifn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=bcn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=ifn1_link1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n01] => (item=bcn1_bond1)
[2026-09-03T04:28:09.784Z] changed: [an-a01n02] => (item=ifn1_bridge1)
[2026-09-03T04:28:09.784Z] changed: [an-a01dr01] => (item=bcn1_bridge1)
[2026-09-03T04:28:09.784Z] 
[2026-09-03T04:28:09.784Z] PLAY [check pacemaker status after deployment] *********************************
[2026-09-03T04:28:09.784Z] 
[2026-09-03T04:28:09.784Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:28:10.521Z] ok: [an-a01n01]
[2026-09-03T04:28:10.521Z] ok: [an-a01n02]
[2026-09-03T04:28:10.521Z] 
[2026-09-03T04:28:10.521Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] ***
[2026-09-03T04:28:10.521Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T04:28:10.521Z] 
[2026-09-03T04:28:10.521Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T04:28:10.521Z] ] ***
[2026-09-03T04:28:14.397Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T04:28:14.397Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T04:28:14.397Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T04:28:14.397Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T04:28:14.397Z] 
[2026-09-03T04:28:14.397Z] PLAY [check drbd kmod] *********************************************************
[2026-09-03T04:28:14.397Z] 
[2026-09-03T04:28:14.397Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:28:16.109Z] ok: [an-a01n01]
[2026-09-03T04:28:16.109Z] ok: [an-a01n02]
[2026-09-03T04:28:16.109Z] ok: [an-a01dr01]
[2026-09-03T04:28:16.109Z] 
[2026-09-03T04:28:16.109Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd
[2026-09-03T04:28:16.109Z] ] ***
[2026-09-03T04:28:16.844Z] ok: [an-a01n01]
[2026-09-03T04:28:16.845Z] ok: [an-a01dr01]
[2026-09-03T04:28:16.845Z] ok: [an-a01n02]
[2026-09-03T04:28:16.845Z] 
[2026-09-03T04:28:16.845Z] TASK [load drbd kernel module _raw_params=modprobe drbd
[2026-09-03T04:28:16.845Z] ] ***********************
[2026-09-03T04:28:16.845Z] skipping: [an-a01n01]
[2026-09-03T04:28:16.845Z] skipping: [an-a01n02]
[2026-09-03T04:28:16.845Z] skipping: [an-a01dr01]
[2026-09-03T04:28:16.845Z] 
[2026-09-03T04:28:16.845Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd
[2026-09-03T04:28:16.845Z] ] ***
[2026-09-03T04:28:16.845Z] skipping: [an-a01n01]
[2026-09-03T04:28:17.187Z] skipping: [an-a01n02]
[2026-09-03T04:28:17.187Z] skipping: [an-a01dr01]
[2026-09-03T04:28:17.187Z] 
[2026-09-03T04:28:17.187Z] PLAY [Copy iso images to striker01] ********************************************
[2026-09-03T04:28:17.187Z] 
[2026-09-03T04:28:17.187Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:28:19.508Z] ok: [an-striker01]
[2026-09-03T04:28:19.508Z] 
[2026-09-03T04:28:19.508Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] *******************
[2026-09-03T04:28:20.246Z] ok: [an-striker01]
[2026-09-03T04:28:20.246Z] 
[2026-09-03T04:28:20.246Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ******
[2026-09-03T04:28:21.434Z] changed: [an-striker01]
[2026-09-03T04:28:21.434Z] 
[2026-09-03T04:28:21.434Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] ***
[2026-09-03T04:28:32.420Z] changed: [an-striker01] => (item=deploy1.iso)
[2026-09-03T04:28:32.420Z] changed: [an-striker01] => (item=deploy2.iso)
[2026-09-03T04:28:32.420Z] changed: [an-striker01] => (item=deploy3.iso)
[2026-09-03T04:28:32.420Z] changed: [an-striker01] => (item=deploy4.iso)
[2026-09-03T04:28:32.420Z] changed: [an-striker01] => (item=deploy5.iso)
[2026-09-03T04:28:32.420Z] 
[2026-09-03T04:28:32.420Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/
[2026-09-03T04:28:32.420Z] ] ***
[2026-09-03T04:28:32.420Z] changed: [an-striker01]
[2026-09-03T04:28:32.420Z] 
[2026-09-03T04:28:32.420Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] *******************
[2026-09-03T04:28:33.156Z] changed: [an-striker01]
[2026-09-03T04:28:33.156Z] 
[2026-09-03T04:28:33.156Z] PLAY [Check iso images propagation] ********************************************
[2026-09-03T04:28:33.156Z] 
[2026-09-03T04:28:33.156Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:28:35.481Z] ok: [an-a01n02]
[2026-09-03T04:28:35.481Z] ok: [an-a01n01]
[2026-09-03T04:28:35.481Z] ok: [an-a01dr01]
[2026-09-03T04:28:35.481Z] ok: [an-striker01]
[2026-09-03T04:28:35.481Z] ok: [an-striker02]
[2026-09-03T04:28:35.481Z] 
[2026-09-03T04:28:35.481Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then
[2026-09-03T04:28:35.481Z]  exit 1
[2026-09-03T04:28:35.481Z] else
[2026-09-03T04:28:35.481Z]  exit 0
[2026-09-03T04:28:35.481Z] fi
[2026-09-03T04:28:35.481Z] ] ***
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-03T04:30:13.947Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-03T04:30:13.947Z] ok: [an-a01n01]
[2026-09-03T04:30:13.947Z] ok: [an-a01n02]
[2026-09-03T04:30:13.947Z] ok: [an-a01dr01]
[2026-09-03T04:30:13.947Z] ok: [an-striker01]
[2026-09-03T04:30:13.947Z] ok: [an-striker02]
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] PLAY [example tests] ***********************************************************
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:13.947Z] ok: [an-a01n01]
[2026-09-03T04:30:13.947Z] ok: [an-a01n02]
[2026-09-03T04:30:13.947Z] ok: [an-a01dr01]
[2026-09-03T04:30:13.947Z] ok: [an-striker01]
[2026-09-03T04:30:13.947Z] ok: [an-striker02]
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true
[2026-09-03T04:30:13.947Z] ] ***
[2026-09-03T04:30:13.947Z] skipping: [an-a01n01]
[2026-09-03T04:30:13.947Z] skipping: [an-a01n02]
[2026-09-03T04:30:13.947Z] skipping: [an-a01dr01]
[2026-09-03T04:30:13.947Z] ok: [an-striker02]
[2026-09-03T04:30:13.947Z] ok: [an-striker01]
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true
[2026-09-03T04:30:13.947Z] ] ***
[2026-09-03T04:30:13.947Z] skipping: [an-striker01]
[2026-09-03T04:30:13.947Z] skipping: [an-striker02]
[2026-09-03T04:30:13.947Z] ok: [an-a01n01]
[2026-09-03T04:30:13.947Z] ok: [an-a01n02]
[2026-09-03T04:30:13.947Z] ok: [an-a01dr01]
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] TASK [inject failure _raw_params=/usr/bin/false
[2026-09-03T04:30:13.947Z] ] *******************************
[2026-09-03T04:30:13.947Z] skipping: [an-striker01]
[2026-09-03T04:30:13.947Z] skipping: [an-striker02]
[2026-09-03T04:30:13.947Z] skipping: [an-a01n01]
[2026-09-03T04:30:13.947Z] skipping: [an-a01n02]
[2026-09-03T04:30:13.947Z] skipping: [an-a01dr01]
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] PLAY [Record anvil available resources from striker01] *************************
[2026-09-03T04:30:13.947Z] 
[2026-09-03T04:30:13.947Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:14.291Z] ok: [an-striker01]
[2026-09-03T04:30:14.291Z] 
[2026-09-03T04:30:14.291Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start
[2026-09-03T04:30:14.291Z] ] ***
[2026-09-03T04:30:19.148Z] ok: [an-striker01]
[2026-09-03T04:30:19.148Z] 
[2026-09-03T04:30:19.148Z] PLAY [check selinux status] ****************************************************
[2026-09-03T04:30:19.148Z] 
[2026-09-03T04:30:19.148Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:22.183Z] ok: [an-a01n01]
[2026-09-03T04:30:22.183Z] ok: [an-a01dr01]
[2026-09-03T04:30:22.183Z] ok: [an-a01n02]
[2026-09-03T04:30:22.183Z] ok: [an-striker01]
[2026-09-03T04:30:22.183Z] ok: [an-striker02]
[2026-09-03T04:30:22.183Z] 
[2026-09-03T04:30:22.183Z] TASK [check selinux status permissive (8) var=ansible_selinux] *****************
[2026-09-03T04:30:22.183Z] skipping: [an-striker01]
[2026-09-03T04:30:22.183Z] skipping: [an-striker02]
[2026-09-03T04:30:22.183Z] skipping: [an-a01n01]
[2026-09-03T04:30:22.183Z] skipping: [an-a01n02]
[2026-09-03T04:30:22.183Z] skipping: [an-a01dr01]
[2026-09-03T04:30:22.183Z] 
[2026-09-03T04:30:22.183Z] TASK [check selinux status permissive (9) var=ansible_selinux] *****************
[2026-09-03T04:30:22.183Z] ok: [an-striker01] => {
[2026-09-03T04:30:22.183Z]     "ansible_selinux": {
[2026-09-03T04:30:22.183Z]         "config_mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "policyvers": 33,
[2026-09-03T04:30:22.183Z]         "status": "enabled",
[2026-09-03T04:30:22.183Z]         "type": "targeted"
[2026-09-03T04:30:22.183Z]     },
[2026-09-03T04:30:22.183Z]     "failed_when_result": false
[2026-09-03T04:30:22.183Z] }
[2026-09-03T04:30:22.183Z] ok: [an-striker02] => {
[2026-09-03T04:30:22.183Z]     "ansible_selinux": {
[2026-09-03T04:30:22.183Z]         "config_mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "policyvers": 33,
[2026-09-03T04:30:22.183Z]         "status": "enabled",
[2026-09-03T04:30:22.183Z]         "type": "targeted"
[2026-09-03T04:30:22.183Z]     },
[2026-09-03T04:30:22.183Z]     "failed_when_result": false
[2026-09-03T04:30:22.183Z] }
[2026-09-03T04:30:22.183Z] ok: [an-a01n01] => {
[2026-09-03T04:30:22.183Z]     "ansible_selinux": {
[2026-09-03T04:30:22.183Z]         "config_mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "policyvers": 33,
[2026-09-03T04:30:22.183Z]         "status": "enabled",
[2026-09-03T04:30:22.183Z]         "type": "targeted"
[2026-09-03T04:30:22.183Z]     },
[2026-09-03T04:30:22.183Z]     "failed_when_result": false
[2026-09-03T04:30:22.183Z] }
[2026-09-03T04:30:22.183Z] ok: [an-a01n02] => {
[2026-09-03T04:30:22.183Z]     "ansible_selinux": {
[2026-09-03T04:30:22.183Z]         "config_mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "policyvers": 33,
[2026-09-03T04:30:22.183Z]         "status": "enabled",
[2026-09-03T04:30:22.183Z]         "type": "targeted"
[2026-09-03T04:30:22.183Z]     },
[2026-09-03T04:30:22.183Z]     "failed_when_result": false
[2026-09-03T04:30:22.183Z] }
[2026-09-03T04:30:22.183Z] ok: [an-a01dr01] => {
[2026-09-03T04:30:22.183Z]     "ansible_selinux": {
[2026-09-03T04:30:22.183Z]         "config_mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "mode": "enforcing",
[2026-09-03T04:30:22.183Z]         "policyvers": 33,
[2026-09-03T04:30:22.183Z]         "status": "enabled",
[2026-09-03T04:30:22.183Z]         "type": "targeted"
[2026-09-03T04:30:22.183Z]     },
[2026-09-03T04:30:22.183Z]     "failed_when_result": false
[2026-09-03T04:30:22.183Z] }
[2026-09-03T04:30:22.183Z] 
[2026-09-03T04:30:22.183Z] PLAY [Create random file on striker01] *****************************************
[2026-09-03T04:30:22.183Z] 
[2026-09-03T04:30:22.183Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:24.506Z] ok: [an-striker01]
[2026-09-03T04:30:24.506Z] 
[2026-09-03T04:30:24.506Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] ***
[2026-09-03T04:30:26.219Z] changed: [an-striker01]
[2026-09-03T04:30:26.219Z] 
[2026-09-03T04:30:26.219Z] PLAY [Check archive file propagation] ******************************************
[2026-09-03T04:30:26.219Z] 
[2026-09-03T04:30:26.219Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:28.569Z] ok: [an-a01n02]
[2026-09-03T04:30:28.569Z] ok: [an-a01n01]
[2026-09-03T04:30:28.569Z] ok: [an-a01dr01]
[2026-09-03T04:30:28.569Z] ok: [an-striker01]
[2026-09-03T04:30:28.569Z] ok: [an-striker02]
[2026-09-03T04:30:28.569Z] 
[2026-09-03T04:30:28.569Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then
[2026-09-03T04:30:28.569Z]  exit 1
[2026-09-03T04:30:28.569Z] else
[2026-09-03T04:30:28.569Z]  exit 0
[2026-09-03T04:30:28.569Z] fi
[2026-09-03T04:30:28.569Z] ] ***
[2026-09-03T04:30:51.980Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:51.980Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:51.980Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:51.980Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-03T04:30:51.980Z] ok: [an-striker01]
[2026-09-03T04:30:51.980Z] ok: [an-a01n01]
[2026-09-03T04:30:51.980Z] ok: [an-a01n02]
[2026-09-03T04:30:51.980Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left).
[2026-09-03T04:30:51.980Z] ok: [an-striker02]
[2026-09-03T04:30:51.980Z] ok: [an-a01dr01]
[2026-09-03T04:30:51.980Z] 
[2026-09-03T04:30:51.980Z] PLAY [test fencing] ************************************************************
[2026-09-03T04:30:51.980Z] 
[2026-09-03T04:30:51.980Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:30:52.323Z] ok: [an-a01n01]
[2026-09-03T04:30:52.323Z] ok: [an-a01n02]
[2026-09-03T04:30:52.323Z] 
[2026-09-03T04:30:52.323Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T04:30:52.665Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T04:30:52.665Z] 
[2026-09-03T04:30:52.665Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T04:30:52.665Z] ] ***
[2026-09-03T04:30:56.542Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T04:30:56.542Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T04:30:56.542Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T04:30:56.542Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T04:30:56.542Z] 
[2026-09-03T04:30:56.542Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************
[2026-09-03T04:30:56.542Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T04:30:56.542Z] 
[2026-09-03T04:30:56.542Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T04:30:56.542Z] ] **********
[2026-09-03T04:30:57.277Z] skipping: [an-a01n02]
[2026-09-03T04:30:57.277Z] changed: [an-a01n01]
[2026-09-03T04:30:57.277Z] 
[2026-09-03T04:30:57.277Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T04:30:57.277Z] skipping: [an-a01n02]
[2026-09-03T04:30:57.277Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01
[2026-09-03T04:30:57.277Z] 
[2026-09-03T04:30:57.277Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T04:34:21.522Z] ok: [an-a01n01 -> localhost]
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T04:34:21.522Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T04:34:21.522Z] ] ***
[2026-09-03T04:34:21.522Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T04:34:21.522Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T04:34:21.522Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T04:34:21.522Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************
[2026-09-03T04:34:21.522Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T04:34:21.522Z] ] **********
[2026-09-03T04:34:21.522Z] skipping: [an-a01n01]
[2026-09-03T04:34:21.522Z] changed: [an-a01n02]
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T04:34:21.522Z] skipping: [an-a01n01]
[2026-09-03T04:34:21.522Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02
[2026-09-03T04:34:21.522Z] 
[2026-09-03T04:34:21.522Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T04:37:30.561Z] ok: [an-a01n02 -> localhost]
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T04:37:30.561Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T04:37:30.561Z] ] ***
[2026-09-03T04:37:30.561Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T04:37:30.561Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T04:37:30.561Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T04:37:30.561Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] PLAY [advanced server tests] ***************************************************
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T04:37:30.561Z] ok: [an-a01n01]
[2026-09-03T04:37:30.561Z] ok: [an-a01n02]
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T04:37:30.561Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T04:37:30.561Z] 
[2026-09-03T04:37:30.561Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T04:37:30.561Z] ] ***
[2026-09-03T04:37:33.598Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T04:37:33.598Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T04:37:33.598Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T04:37:33.598Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T04:37:33.598Z] 
[2026-09-03T04:37:33.598Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-03T04:37:33.598Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-03T04:37:33.598Z] 
[2026-09-03T04:37:33.598Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-03T04:37:33.598Z] if [ -z "$storage" ]; then
[2026-09-03T04:37:33.598Z]   echo "Unable to detect shared storage"
[2026-09-03T04:37:33.598Z]   exit 1
[2026-09-03T04:37:33.598Z] fi
[2026-09-03T04:37:33.598Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T04:37:33.598Z] ] ***
[2026-09-03T04:37:39.627Z] changed: [an-a01n01]
[2026-09-03T04:37:39.627Z] changed: [an-a01n02]
[2026-09-03T04:37:39.627Z] 
[2026-09-03T04:37:39.627Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-03T04:37:39.627Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-03T04:37:39.627Z] 
[2026-09-03T04:37:39.627Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-03T04:37:39.627Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-03T04:37:39.627Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-03T04:37:39.627Z]   echo "Unable to detect all deploy iso"
[2026-09-03T04:37:39.627Z]   echo "Detected isos"
[2026-09-03T04:37:39.627Z]   echo "$isos"
[2026-09-03T04:37:39.627Z]   exit 1
[2026-09-03T04:37:39.627Z] fi
[2026-09-03T04:37:39.627Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-03T04:37:39.627Z] if [ -z "$installiso" ]; then
[2026-09-03T04:37:39.627Z]   echo "Unable to detect install iso"
[2026-09-03T04:37:39.627Z]   echo "Detected isos"
[2026-09-03T04:37:39.627Z]   echo "$isos"
[2026-09-03T04:37:39.627Z]   exit 1
[2026-09-03T04:37:39.627Z] fi
[2026-09-03T04:37:39.627Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T04:37:39.627Z] ] ***
[2026-09-03T04:37:45.770Z] changed: [an-a01n02]
[2026-09-03T04:37:45.770Z] changed: [an-a01n01]
[2026-09-03T04:37:45.770Z] 
[2026-09-03T04:37:45.770Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] ***************
[2026-09-03T04:37:45.770Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T04:37:45.770Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T04:37:45.770Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T04:37:45.770Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T04:37:45.770Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T04:37:45.770Z] 
[2026-09-03T04:37:45.770Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T04:37:45.770Z] ] ***
[2026-09-03T04:37:46.959Z] skipping: [an-a01n02]
[2026-09-03T04:37:46.959Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.959337", "end": "2026-09-03 00:37:46.534499", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 00:37:45.575162", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T04:37:46.959Z] ...ignoring
[2026-09-03T04:37:46.959Z] 
[2026-09-03T04:37:46.959Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T04:37:46.959Z]   --ci-test \
[2026-09-03T04:37:46.959Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T04:37:46.959Z]   --os almalinux9 \
[2026-09-03T04:37:46.959Z]   --cpu 2 --ram 4G \
[2026-09-03T04:37:46.959Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T04:37:46.959Z]   --install-media {{ installisouuid }} \
[2026-09-03T04:37:46.959Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T04:37:46.959Z] ] ***
[2026-09-03T04:37:55.999Z] skipping: [an-a01n02]
[2026-09-03T04:37:55.999Z] changed: [an-a01n01]
[2026-09-03T04:37:55.999Z] 
[2026-09-03T04:37:55.999Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T04:37:55.999Z] skipping: [an-a01n01]
[2026-09-03T04:37:55.999Z] skipping: [an-a01n02]
[2026-09-03T04:37:55.999Z] 
[2026-09-03T04:37:55.999Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T04:37:55.999Z] skipping: [an-a01n01]
[2026-09-03T04:37:55.999Z] skipping: [an-a01n02]
[2026-09-03T04:37:55.999Z] 
[2026-09-03T04:37:55.999Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T04:37:55.999Z] skipping: [an-a01n01]
[2026-09-03T04:37:55.999Z] skipping: [an-a01n02]
[2026-09-03T04:37:55.999Z] 
[2026-09-03T04:37:55.999Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T04:37:55.999Z] skipping: [an-a01n01]
[2026-09-03T04:37:55.999Z] skipping: [an-a01n02]
[2026-09-03T04:37:55.999Z] 
[2026-09-03T04:37:55.999Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T04:37:55.999Z] ] ***
[2026-09-03T04:37:57.768Z] skipping: [an-a01n02]
[2026-09-03T04:37:57.768Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.907619", "end": "2026-09-03 00:37:57.429212", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 00:37:56.521593", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T04:37:57.768Z] ...ignoring
[2026-09-03T04:37:57.768Z] 
[2026-09-03T04:37:57.768Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T04:37:57.768Z]   --ci-test \
[2026-09-03T04:37:57.768Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T04:37:57.768Z]   --os almalinux9 \
[2026-09-03T04:37:57.768Z]   --cpu 2 --ram 4G \
[2026-09-03T04:37:57.768Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T04:37:57.768Z]   --install-media {{ installisouuid }} \
[2026-09-03T04:37:57.768Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T04:37:57.768Z] ] ***
[2026-09-03T04:38:06.806Z] skipping: [an-a01n02]
[2026-09-03T04:38:06.806Z] changed: [an-a01n01]
[2026-09-03T04:38:06.806Z] 
[2026-09-03T04:38:06.806Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T04:38:06.806Z] skipping: [an-a01n01]
[2026-09-03T04:38:06.806Z] skipping: [an-a01n02]
[2026-09-03T04:38:06.806Z] 
[2026-09-03T04:38:06.806Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T04:38:06.806Z] skipping: [an-a01n01]
[2026-09-03T04:38:06.806Z] skipping: [an-a01n02]
[2026-09-03T04:38:06.806Z] 
[2026-09-03T04:38:06.806Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T04:38:06.806Z] skipping: [an-a01n01]
[2026-09-03T04:38:06.806Z] skipping: [an-a01n02]
[2026-09-03T04:38:06.806Z] 
[2026-09-03T04:38:06.806Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T04:38:07.148Z] skipping: [an-a01n01]
[2026-09-03T04:38:07.148Z] skipping: [an-a01n02]
[2026-09-03T04:38:07.148Z] 
[2026-09-03T04:38:07.148Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T04:38:07.148Z] ] ***
[2026-09-03T04:38:08.859Z] skipping: [an-a01n02]
[2026-09-03T04:38:08.859Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:01.015091", "end": "2026-09-03 00:38:08.381116", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 00:38:07.366025", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T04:38:08.859Z] ...ignoring
[2026-09-03T04:38:08.859Z] 
[2026-09-03T04:38:08.859Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T04:38:08.859Z]   --ci-test \
[2026-09-03T04:38:08.859Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T04:38:08.859Z]   --os almalinux9 \
[2026-09-03T04:38:08.859Z]   --cpu 2 --ram 4G \
[2026-09-03T04:38:08.859Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T04:38:08.859Z]   --install-media {{ installisouuid }} \
[2026-09-03T04:38:08.859Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T04:38:08.859Z] ] ***
[2026-09-03T04:38:17.900Z] skipping: [an-a01n02]
[2026-09-03T04:38:17.900Z] changed: [an-a01n01]
[2026-09-03T04:38:17.900Z] 
[2026-09-03T04:38:17.900Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T04:38:17.900Z] skipping: [an-a01n01]
[2026-09-03T04:38:17.900Z] skipping: [an-a01n02]
[2026-09-03T04:38:17.900Z] 
[2026-09-03T04:38:17.900Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T04:38:17.900Z] skipping: [an-a01n01]
[2026-09-03T04:38:17.900Z] skipping: [an-a01n02]
[2026-09-03T04:38:17.900Z] 
[2026-09-03T04:38:17.900Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T04:38:17.900Z] skipping: [an-a01n01]
[2026-09-03T04:38:17.900Z] skipping: [an-a01n02]
[2026-09-03T04:38:17.900Z] 
[2026-09-03T04:38:17.900Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T04:38:18.242Z] skipping: [an-a01n01]
[2026-09-03T04:38:18.242Z] skipping: [an-a01n02]
[2026-09-03T04:38:18.242Z] 
[2026-09-03T04:38:18.242Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T04:38:18.242Z] ] ***
[2026-09-03T04:38:19.955Z] skipping: [an-a01n02]
[2026-09-03T04:38:19.955Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.988003", "end": "2026-09-03 00:38:19.470850", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 00:38:18.482847", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T04:38:19.955Z] ...ignoring
[2026-09-03T04:38:19.955Z] 
[2026-09-03T04:38:19.955Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T04:38:19.955Z]   --ci-test \
[2026-09-03T04:38:19.955Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T04:38:19.955Z]   --os almalinux9 \
[2026-09-03T04:38:19.955Z]   --cpu 2 --ram 4G \
[2026-09-03T04:38:19.955Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T04:38:19.955Z]   --install-media {{ installisouuid }} \
[2026-09-03T04:38:19.955Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T04:38:19.955Z] ] ***
[2026-09-03T04:38:28.997Z] skipping: [an-a01n02]
[2026-09-03T04:38:28.997Z] changed: [an-a01n01]
[2026-09-03T04:38:28.997Z] 
[2026-09-03T04:38:28.997Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T04:38:28.997Z] skipping: [an-a01n01]
[2026-09-03T04:38:28.997Z] skipping: [an-a01n02]
[2026-09-03T04:38:28.997Z] 
[2026-09-03T04:38:28.997Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T04:38:29.340Z] skipping: [an-a01n01]
[2026-09-03T04:38:29.340Z] skipping: [an-a01n02]
[2026-09-03T04:38:29.340Z] 
[2026-09-03T04:38:29.340Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T04:38:29.340Z] skipping: [an-a01n01]
[2026-09-03T04:38:29.340Z] skipping: [an-a01n02]
[2026-09-03T04:38:29.340Z] 
[2026-09-03T04:38:29.340Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T04:38:29.340Z] skipping: [an-a01n01]
[2026-09-03T04:38:29.340Z] skipping: [an-a01n02]
[2026-09-03T04:38:29.340Z] 
[2026-09-03T04:38:29.340Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T04:38:29.340Z] ] ***
[2026-09-03T04:38:31.053Z] skipping: [an-a01n02]
[2026-09-03T04:38:31.053Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.961673", "end": "2026-09-03 00:38:30.745930", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 00:38:29.784257", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T04:38:31.053Z] ...ignoring
[2026-09-03T04:38:31.053Z] 
[2026-09-03T04:38:31.053Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T04:38:31.053Z]   --ci-test \
[2026-09-03T04:38:31.053Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T04:38:31.053Z]   --os almalinux9 \
[2026-09-03T04:38:31.053Z]   --cpu 2 --ram 4G \
[2026-09-03T04:38:31.053Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T04:38:31.053Z]   --install-media {{ installisouuid }} \
[2026-09-03T04:38:31.053Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T04:38:31.053Z] ] ***
[2026-09-03T04:38:44.348Z] skipping: [an-a01n02]
[2026-09-03T04:38:44.348Z] changed: [an-a01n01]
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T04:38:44.348Z] skipping: [an-a01n01]
[2026-09-03T04:38:44.348Z] skipping: [an-a01n02]
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T04:38:44.348Z] skipping: [an-a01n01]
[2026-09-03T04:38:44.348Z] skipping: [an-a01n02]
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T04:38:44.348Z] skipping: [an-a01n01]
[2026-09-03T04:38:44.348Z] skipping: [an-a01n02]
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T04:38:44.348Z] skipping: [an-a01n01]
[2026-09-03T04:38:44.348Z] skipping: [an-a01n02]
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] ***
[2026-09-03T04:38:44.348Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T04:38:44.348Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T04:38:44.348Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T04:38:44.348Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T04:38:44.348Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T04:38:44.348Z] 
[2026-09-03T04:38:44.348Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T04:38:44.348Z] ] ***
[2026-09-03T04:39:12.323Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left).
[2026-09-03T04:39:12.323Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left).
[2026-09-03T04:39:12.323Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left).
[2026-09-03T04:39:12.323Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left).
[2026-09-03T04:39:12.323Z] ok: [an-a01n02]
[2026-09-03T04:39:12.323Z] ok: [an-a01n01]
[2026-09-03T04:39:12.323Z] 
[2026-09-03T04:39:12.323Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T04:39:12.323Z] ] ***
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left).
[2026-09-03T04:41:21.251Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left).
[2026-09-03T04:41:21.251Z] ok: [an-a01n02]
[2026-09-03T04:41:21.251Z] ok: [an-a01n01]
[2026-09-03T04:41:21.251Z] 
[2026-09-03T04:41:21.251Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T04:41:21.251Z] ] ***
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left).
[2026-09-03T04:44:00.086Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left).
[2026-09-03T04:44:00.086Z] ok: [an-a01n02]
[2026-09-03T04:44:00.086Z] ok: [an-a01n01]
[2026-09-03T04:44:00.086Z] 
[2026-09-03T04:44:00.086Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T04:44:00.086Z] ] ***
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left).
[2026-09-03T04:46:54.044Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left).
[2026-09-03T04:46:54.044Z] ok: [an-a01n02]
[2026-09-03T04:46:54.044Z] ok: [an-a01n01]
[2026-09-03T04:46:54.044Z] 
[2026-09-03T04:46:54.044Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T04:46:54.044Z] ] ***
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (344 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (344 retries left).
[2026-09-03T04:50:18.278Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (343 retries left).
[2026-09-03T04:50:18.278Z] ok: [an-a01n01]
[2026-09-03T04:50:18.278Z] ok: [an-a01n02]
[2026-09-03T04:50:18.278Z] 
[2026-09-03T04:50:18.278Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] ***
[2026-09-03T04:50:18.278Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=1)
[2026-09-03T04:50:18.278Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=2)
[2026-09-03T04:50:18.278Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=3)
[2026-09-03T04:50:18.278Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=4)
[2026-09-03T04:50:18.278Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=5)
[2026-09-03T04:50:18.278Z] 
[2026-09-03T04:50:18.278Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T04:50:18.278Z] ] ***
[2026-09-03T04:50:22.150Z] ok: [an-a01n02]
[2026-09-03T04:50:22.150Z] ok: [an-a01n01]
[2026-09-03T04:50:22.150Z] 
[2026-09-03T04:50:22.150Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T04:50:22.150Z] ] ***
[2026-09-03T04:50:23.336Z] ok: [an-a01n02]
[2026-09-03T04:50:23.336Z] ok: [an-a01n01]
[2026-09-03T04:50:23.336Z] 
[2026-09-03T04:50:23.336Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T04:50:23.336Z] ] ***
[2026-09-03T04:50:24.522Z] ok: [an-a01n02]
[2026-09-03T04:50:24.522Z] ok: [an-a01n01]
[2026-09-03T04:50:24.522Z] 
[2026-09-03T04:50:24.522Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T04:50:24.522Z] ] ***
[2026-09-03T04:50:25.256Z] ok: [an-a01n02]
[2026-09-03T04:50:25.256Z] ok: [an-a01n01]
[2026-09-03T04:50:25.256Z] 
[2026-09-03T04:50:25.256Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T04:50:25.256Z] ] ***
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left).
[2026-09-03T04:52:18.939Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left).
[2026-09-03T04:52:18.939Z] ok: [an-a01n02]
[2026-09-03T04:52:18.939Z] ok: [an-a01n01]
[2026-09-03T04:52:18.939Z] 
[2026-09-03T04:52:18.939Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] ***
[2026-09-03T04:52:18.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T04:52:18.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T04:52:18.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T04:52:18.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T04:52:18.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T04:52:18.939Z] 
[2026-09-03T04:52:18.939Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T04:52:18.939Z] ] ***
[2026-09-03T04:52:18.939Z] ok: [an-a01n02]
[2026-09-03T04:52:18.939Z] ok: [an-a01n01]
[2026-09-03T04:52:18.939Z] 
[2026-09-03T04:52:18.939Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T04:52:18.939Z] ] ***
[2026-09-03T04:52:18.939Z] ok: [an-a01n02]
[2026-09-03T04:52:18.939Z] ok: [an-a01n01]
[2026-09-03T04:52:18.939Z] 
[2026-09-03T04:52:18.939Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T04:52:18.939Z] ] ***
[2026-09-03T04:52:18.939Z] ok: [an-a01n02]
[2026-09-03T04:52:18.939Z] ok: [an-a01n01]
[2026-09-03T04:52:18.939Z] 
[2026-09-03T04:52:18.939Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T04:52:18.939Z] ] ***
[2026-09-03T04:54:57.874Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left).
[2026-09-03T04:54:57.875Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left).
[2026-09-03T04:54:57.875Z] ok: [an-a01n02]
[2026-09-03T04:54:57.875Z] ok: [an-a01n01]
[2026-09-03T04:54:57.875Z] 
[2026-09-03T04:54:57.875Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T04:54:57.875Z] ] ***
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left).
[2026-09-03T04:59:07.276Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left).
[2026-09-03T04:59:07.276Z] ok: [an-a01n02]
[2026-09-03T04:59:07.276Z] ok: [an-a01n01]
[2026-09-03T04:59:07.276Z] 
[2026-09-03T04:59:07.276Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] ***
[2026-09-03T04:59:07.276Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T04:59:07.276Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T04:59:07.276Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T04:59:07.276Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T04:59:07.276Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T04:59:07.276Z] 
[2026-09-03T04:59:07.276Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T04:59:07.276Z] ] ***
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (360 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (360 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (359 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (359 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (358 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (358 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (357 retries left).
[2026-09-03T05:00:16.844Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to respond to ssh (357 retries left).
[2026-09-03T05:00:16.844Z] ok: [an-a01n01]
[2026-09-03T05:00:16.844Z] ok: [an-a01n02]
[2026-09-03T05:00:16.844Z] 
[2026-09-03T05:00:16.844Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:00:16.844Z] ] ***
[2026-09-03T05:00:44.906Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left).
[2026-09-03T05:00:44.906Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left).
[2026-09-03T05:00:44.906Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left).
[2026-09-03T05:00:44.906Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left).
[2026-09-03T05:00:44.906Z] ok: [an-a01n02]
[2026-09-03T05:00:44.906Z] ok: [an-a01n01]
[2026-09-03T05:00:44.906Z] 
[2026-09-03T05:00:44.906Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:00:44.906Z] ] ***
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left).
[2026-09-03T05:02:38.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left).
[2026-09-03T05:02:38.495Z] ok: [an-a01n01]
[2026-09-03T05:02:38.495Z] ok: [an-a01n02]
[2026-09-03T05:02:38.495Z] 
[2026-09-03T05:02:38.495Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:02:38.495Z] ] ***
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left).
[2026-09-03T05:05:17.313Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left).
[2026-09-03T05:05:17.313Z] ok: [an-a01n02]
[2026-09-03T05:05:17.313Z] ok: [an-a01n01]
[2026-09-03T05:05:17.313Z] 
[2026-09-03T05:05:17.313Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:05:17.313Z] ] ***
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left).
[2026-09-03T05:08:26.434Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left).
[2026-09-03T05:08:26.434Z] ok: [an-a01n02]
[2026-09-03T05:08:26.434Z] ok: [an-a01n01]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] ***
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ******************
[2026-09-03T05:08:26.434Z] ok: [an-a01n01]
[2026-09-03T05:08:26.434Z] ok: [an-a01n02]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:08:26.434Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:08:26.434Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:08:26.434Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:08:26.434Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:08:26.434Z]   echo "pcs:"
[2026-09-03T05:08:26.434Z]   echo "$pcsstatus"
[2026-09-03T05:08:26.434Z]   exit 1
[2026-09-03T05:08:26.434Z] fi
[2026-09-03T05:08:26.434Z] exit 0
[2026-09-03T05:08:26.434Z] ] ***
[2026-09-03T05:08:26.434Z] ok: [an-a01n02]
[2026-09-03T05:08:26.434Z] ok: [an-a01n01]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:08:26.434Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:08:26.434Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:08:26.434Z] ret=0
[2026-09-03T05:08:26.434Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:08:26.434Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:08:26.434Z]   echo "$pcsstatus"
[2026-09-03T05:08:26.434Z]   ret=1
[2026-09-03T05:08:26.434Z] fi
[2026-09-03T05:08:26.434Z] echo "$hypervisor"
[2026-09-03T05:08:26.434Z] exit $ret
[2026-09-03T05:08:26.434Z] ] ***
[2026-09-03T05:08:26.434Z] ok: [an-a01n02]
[2026-09-03T05:08:26.434Z] ok: [an-a01n01]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-03T05:08:26.434Z] ok: [an-a01n01]
[2026-09-03T05:08:26.434Z] ok: [an-a01n02]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T05:08:26.434Z] ] **********
[2026-09-03T05:08:26.434Z] skipping: [an-a01n02]
[2026-09-03T05:08:26.434Z] changed: [an-a01n01]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T05:08:26.434Z] skipping: [an-a01n01]
[2026-09-03T05:08:26.434Z] skipping: [an-a01n02]
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-03T05:08:26.434Z] skipping: [an-a01n01]
[2026-09-03T05:08:26.434Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-03T05:08:26.434Z] 
[2026-09-03T05:08:26.434Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-03T05:08:26.434Z] ] ***
[2026-09-03T05:09:35.999Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-03T05:09:35.999Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-03T05:09:35.999Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-03T05:09:35.999Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-03T05:09:35.999Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-03T05:09:35.999Z] ok: [an-a01n02]
[2026-09-03T05:09:35.999Z] 
[2026-09-03T05:09:35.999Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:09:35.999Z] skipping: [an-a01n01]
[2026-09-03T05:09:35.999Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-03T05:09:35.999Z] 
[2026-09-03T05:09:35.999Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:09:35.999Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:09:35.999Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:09:35.999Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:09:35.999Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:09:35.999Z]   echo "pcs:"
[2026-09-03T05:09:35.999Z]   echo "$pcsstatus"
[2026-09-03T05:09:35.999Z]   exit 1
[2026-09-03T05:09:35.999Z] fi
[2026-09-03T05:09:35.999Z] exit 0
[2026-09-03T05:09:35.999Z] ] ***
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-03T05:10:59.363Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left).
[2026-09-03T05:10:59.363Z] ok: [an-a01n02]
[2026-09-03T05:10:59.363Z] 
[2026-09-03T05:10:59.363Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:10:59.363Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:10:59.363Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:10:59.363Z] ret=0
[2026-09-03T05:10:59.363Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:10:59.363Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:10:59.363Z]   echo "$pcsstatus"
[2026-09-03T05:10:59.363Z]   ret=1
[2026-09-03T05:10:59.363Z] fi
[2026-09-03T05:10:59.363Z] echo "$hypervisor"
[2026-09-03T05:10:59.363Z] exit $ret
[2026-09-03T05:10:59.363Z] ] ***
[2026-09-03T05:10:59.363Z] ok: [an-a01n02]
[2026-09-03T05:10:59.363Z] 
[2026-09-03T05:10:59.363Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01] => (item=1) 
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01] => (item=2) 
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01] => (item=3) 
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01] => (item=4) 
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01] => (item=5) 
[2026-09-03T05:10:59.363Z] skipping: [an-a01n01]
[2026-09-03T05:10:59.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-03T05:10:59.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-03T05:10:59.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-03T05:10:59.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-03T05:10:59.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-03T05:10:59.363Z] 
[2026-09-03T05:10:59.363Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:10:59.363Z] ] ***
[2026-09-03T05:11:22.658Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-03T05:11:22.658Z] ok: [an-a01n02]
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:11:22.658Z] ] ***
[2026-09-03T05:11:22.658Z] ok: [an-a01n02]
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:11:22.658Z] ] ***
[2026-09-03T05:11:22.658Z] ok: [an-a01n02]
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:11:22.658Z] ] ***
[2026-09-03T05:11:22.658Z] ok: [an-a01n02]
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:11:22.658Z] ] ***
[2026-09-03T05:11:22.658Z] ok: [an-a01n02]
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-03T05:11:22.658Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:11:22.658Z] 
[2026-09-03T05:11:22.658Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T05:14:46.840Z] ok: [an-a01n01 -> localhost]
[2026-09-03T05:14:46.840Z] ok: [an-a01n02 -> localhost]
[2026-09-03T05:14:46.840Z] 
[2026-09-03T05:14:46.840Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:14:46.840Z] 
[2026-09-03T05:14:46.840Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:14:46.840Z] ] ***
[2026-09-03T05:14:46.840Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:14:46.840Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:14:46.840Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:14:46.840Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:14:46.840Z] 
[2026-09-03T05:14:46.840Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:14:46.840Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:14:46.840Z] 
[2026-09-03T05:14:46.840Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:14:46.840Z] ] ***
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left).
[2026-09-03T05:18:11.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left).
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ******************
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:18:11.052Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:18:11.052Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:18:11.052Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:18:11.052Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:18:11.052Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:18:11.052Z]   echo "pcs:"
[2026-09-03T05:18:11.052Z]   echo "$pcsstatus"
[2026-09-03T05:18:11.052Z]   exit 1
[2026-09-03T05:18:11.052Z] fi
[2026-09-03T05:18:11.052Z] exit 0
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:18:11.052Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:18:11.052Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:18:11.052Z] ret=0
[2026-09-03T05:18:11.052Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:18:11.052Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:18:11.052Z]   echo "$pcsstatus"
[2026-09-03T05:18:11.052Z]   ret=1
[2026-09-03T05:18:11.052Z] fi
[2026-09-03T05:18:11.052Z] echo "$hypervisor"
[2026-09-03T05:18:11.052Z] exit $ret
[2026-09-03T05:18:11.052Z] ] ***
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-03T05:18:11.052Z] ok: [an-a01n01]
[2026-09-03T05:18:11.052Z] ok: [an-a01n02]
[2026-09-03T05:18:11.052Z] 
[2026-09-03T05:18:11.052Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-03T05:18:11.394Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:18:11.394Z] 
[2026-09-03T05:18:11.394Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T05:18:11.394Z] ] **********
[2026-09-03T05:18:12.127Z] skipping: [an-a01n01]
[2026-09-03T05:18:12.127Z] changed: [an-a01n02]
[2026-09-03T05:18:12.127Z] 
[2026-09-03T05:18:12.127Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T05:18:12.127Z] skipping: [an-a01n01]
[2026-09-03T05:18:12.127Z] skipping: [an-a01n02]
[2026-09-03T05:18:12.127Z] 
[2026-09-03T05:18:12.127Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-03T05:18:12.127Z] skipping: [an-a01n02]
[2026-09-03T05:18:12.127Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01
[2026-09-03T05:18:12.127Z] 
[2026-09-03T05:18:12.127Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-03T05:18:12.127Z] ] ***
[2026-09-03T05:19:21.750Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left).
[2026-09-03T05:19:21.750Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left).
[2026-09-03T05:19:21.750Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left).
[2026-09-03T05:19:21.750Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left).
[2026-09-03T05:19:21.750Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left).
[2026-09-03T05:19:21.750Z] ok: [an-a01n01]
[2026-09-03T05:19:21.750Z] 
[2026-09-03T05:19:21.750Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:19:21.750Z] skipping: [an-a01n02]
[2026-09-03T05:19:21.750Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01
[2026-09-03T05:19:21.750Z] 
[2026-09-03T05:19:21.750Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:19:21.750Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:19:21.750Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:19:21.750Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:19:21.750Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:19:21.750Z]   echo "pcs:"
[2026-09-03T05:19:21.750Z]   echo "$pcsstatus"
[2026-09-03T05:19:21.750Z]   exit 1
[2026-09-03T05:19:21.750Z] fi
[2026-09-03T05:19:21.750Z] exit 0
[2026-09-03T05:19:21.750Z] ] ***
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-03T05:20:45.098Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-03T05:20:45.098Z] ok: [an-a01n01]
[2026-09-03T05:20:45.098Z] 
[2026-09-03T05:20:45.098Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:20:45.098Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:20:45.098Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:20:45.098Z] ret=0
[2026-09-03T05:20:45.098Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:20:45.098Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:20:45.098Z]   echo "$pcsstatus"
[2026-09-03T05:20:45.098Z]   ret=1
[2026-09-03T05:20:45.098Z] fi
[2026-09-03T05:20:45.098Z] echo "$hypervisor"
[2026-09-03T05:20:45.098Z] exit $ret
[2026-09-03T05:20:45.098Z] ] ***
[2026-09-03T05:20:45.098Z] ok: [an-a01n01]
[2026-09-03T05:20:45.098Z] 
[2026-09-03T05:20:45.098Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02] => (item=1) 
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02] => (item=2) 
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02] => (item=3) 
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02] => (item=4) 
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02] => (item=5) 
[2026-09-03T05:20:45.098Z] skipping: [an-a01n02]
[2026-09-03T05:20:45.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1)
[2026-09-03T05:20:45.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2)
[2026-09-03T05:20:45.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3)
[2026-09-03T05:20:45.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4)
[2026-09-03T05:20:45.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5)
[2026-09-03T05:20:45.098Z] 
[2026-09-03T05:20:45.098Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:20:45.098Z] ] ***
[2026-09-03T05:20:50.048Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-03T05:20:50.048Z] ok: [an-a01n01]
[2026-09-03T05:20:50.048Z] 
[2026-09-03T05:20:50.048Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:20:50.048Z] ] ***
[2026-09-03T05:20:51.237Z] ok: [an-a01n01]
[2026-09-03T05:20:51.237Z] 
[2026-09-03T05:20:51.237Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:20:51.237Z] ] ***
[2026-09-03T05:20:52.425Z] ok: [an-a01n01]
[2026-09-03T05:20:52.425Z] 
[2026-09-03T05:20:52.425Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:20:52.425Z] ] ***
[2026-09-03T05:20:53.159Z] ok: [an-a01n01]
[2026-09-03T05:20:53.159Z] 
[2026-09-03T05:20:53.159Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:20:53.159Z] ] ***
[2026-09-03T05:20:54.346Z] ok: [an-a01n01]
[2026-09-03T05:20:54.346Z] 
[2026-09-03T05:20:54.346Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-03T05:20:54.346Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:20:54.346Z] 
[2026-09-03T05:20:54.346Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T05:24:18.566Z] ok: [an-a01n01 -> localhost]
[2026-09-03T05:24:18.566Z] ok: [an-a01n02 -> localhost]
[2026-09-03T05:24:18.566Z] 
[2026-09-03T05:24:18.566Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:24:18.566Z] 
[2026-09-03T05:24:18.566Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:24:18.566Z] ] ***
[2026-09-03T05:24:18.566Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:24:18.566Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:24:18.566Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:24:18.566Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:24:18.566Z] 
[2026-09-03T05:24:18.566Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:24:18.566Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:24:18.566Z] 
[2026-09-03T05:24:18.566Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:24:18.566Z] ] ***
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left).
[2026-09-03T05:25:57.013Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left).
[2026-09-03T05:25:57.013Z] ok: [an-a01n01]
[2026-09-03T05:25:57.013Z] ok: [an-a01n02]
[2026-09-03T05:25:57.013Z] 
[2026-09-03T05:25:57.013Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:25:57.013Z] ] ***
[2026-09-03T05:26:06.836Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-03T05:26:06.836Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-03T05:26:06.836Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-03T05:26:06.836Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-03T05:26:06.836Z] ok: [an-a01n01]
[2026-09-03T05:26:06.836Z] ok: [an-a01n02]
[2026-09-03T05:26:06.836Z] 
[2026-09-03T05:26:06.836Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:26:06.836Z] ] ***
[2026-09-03T05:26:08.030Z] ok: [an-a01n01]
[2026-09-03T05:26:08.030Z] ok: [an-a01n02]
[2026-09-03T05:26:08.030Z] 
[2026-09-03T05:26:08.031Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:26:08.031Z] ] ***
[2026-09-03T05:26:09.749Z] ok: [an-a01n01]
[2026-09-03T05:26:09.749Z] ok: [an-a01n02]
[2026-09-03T05:26:09.749Z] 
[2026-09-03T05:26:09.749Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:26:09.749Z] ] ***
[2026-09-03T05:26:10.960Z] ok: [an-a01n01]
[2026-09-03T05:26:10.960Z] ok: [an-a01n02]
[2026-09-03T05:26:10.960Z] 
[2026-09-03T05:26:10.960Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ******************
[2026-09-03T05:26:10.960Z] ok: [an-a01n01]
[2026-09-03T05:26:10.960Z] ok: [an-a01n02]
[2026-09-03T05:26:10.960Z] 
[2026-09-03T05:26:10.960Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:26:11.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:26:11.304Z] 
[2026-09-03T05:26:11.304Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:26:11.304Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:26:11.304Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:26:11.304Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:26:11.304Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:26:11.304Z]   echo "pcs:"
[2026-09-03T05:26:11.304Z]   echo "$pcsstatus"
[2026-09-03T05:26:11.304Z]   exit 1
[2026-09-03T05:26:11.304Z] fi
[2026-09-03T05:26:11.304Z] exit 0
[2026-09-03T05:26:11.304Z] ] ***
[2026-09-03T05:26:13.628Z] ok: [an-a01n01]
[2026-09-03T05:26:13.628Z] ok: [an-a01n02]
[2026-09-03T05:26:13.628Z] 
[2026-09-03T05:26:13.628Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:26:13.628Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:26:13.628Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:26:13.628Z] ret=0
[2026-09-03T05:26:13.628Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:26:13.628Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:26:13.628Z]   echo "$pcsstatus"
[2026-09-03T05:26:13.628Z]   ret=1
[2026-09-03T05:26:13.628Z] fi
[2026-09-03T05:26:13.628Z] echo "$hypervisor"
[2026-09-03T05:26:13.628Z] exit $ret
[2026-09-03T05:26:13.628Z] ] ***
[2026-09-03T05:26:15.950Z] ok: [an-a01n01]
[2026-09-03T05:26:15.950Z] ok: [an-a01n02]
[2026-09-03T05:26:15.950Z] 
[2026-09-03T05:26:15.950Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-03T05:26:15.950Z] ok: [an-a01n01]
[2026-09-03T05:26:15.950Z] ok: [an-a01n02]
[2026-09-03T05:26:15.950Z] 
[2026-09-03T05:26:15.950Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-03T05:26:16.292Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:26:16.292Z] 
[2026-09-03T05:26:16.292Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T05:26:16.292Z] ] **********
[2026-09-03T05:26:17.029Z] skipping: [an-a01n02]
[2026-09-03T05:26:17.029Z] changed: [an-a01n01]
[2026-09-03T05:26:17.029Z] 
[2026-09-03T05:26:17.029Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T05:26:17.029Z] skipping: [an-a01n01]
[2026-09-03T05:26:17.029Z] skipping: [an-a01n02]
[2026-09-03T05:26:17.029Z] 
[2026-09-03T05:26:17.029Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-03T05:26:17.029Z] skipping: [an-a01n01]
[2026-09-03T05:26:17.029Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-03T05:26:17.029Z] 
[2026-09-03T05:26:17.029Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-03T05:26:17.029Z] ] ***
[2026-09-03T05:27:27.017Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-03T05:27:27.017Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-03T05:27:27.017Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-03T05:27:27.017Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-03T05:27:27.017Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-03T05:27:27.017Z] ok: [an-a01n02]
[2026-09-03T05:27:27.017Z] 
[2026-09-03T05:27:27.017Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:27:27.017Z] skipping: [an-a01n01]
[2026-09-03T05:27:27.017Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-03T05:27:27.017Z] 
[2026-09-03T05:27:27.017Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:27:27.017Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:27:27.017Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:27:27.017Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:27:27.017Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:27:27.017Z]   echo "pcs:"
[2026-09-03T05:27:27.017Z]   echo "$pcsstatus"
[2026-09-03T05:27:27.017Z]   exit 1
[2026-09-03T05:27:27.017Z] fi
[2026-09-03T05:27:27.017Z] exit 0
[2026-09-03T05:27:27.017Z] ] ***
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-03T05:28:36.804Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-03T05:28:36.804Z] ok: [an-a01n02]
[2026-09-03T05:28:36.804Z] 
[2026-09-03T05:28:36.804Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:28:36.804Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:28:36.804Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:28:36.804Z] ret=0
[2026-09-03T05:28:36.804Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:28:36.804Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:28:36.804Z]   echo "$pcsstatus"
[2026-09-03T05:28:36.804Z]   ret=1
[2026-09-03T05:28:36.804Z] fi
[2026-09-03T05:28:36.804Z] echo "$hypervisor"
[2026-09-03T05:28:36.804Z] exit $ret
[2026-09-03T05:28:36.804Z] ] ***
[2026-09-03T05:28:37.990Z] ok: [an-a01n02]
[2026-09-03T05:28:37.990Z] 
[2026-09-03T05:28:37.990Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01] => (item=1) 
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01] => (item=2) 
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01] => (item=3) 
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01] => (item=4) 
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01] => (item=5) 
[2026-09-03T05:28:37.990Z] skipping: [an-a01n01]
[2026-09-03T05:28:37.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-03T05:28:37.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-03T05:28:37.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-03T05:28:37.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-03T05:28:37.990Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-03T05:28:37.990Z] 
[2026-09-03T05:28:37.990Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:28:37.990Z] ] ***
[2026-09-03T05:28:39.179Z] ok: [an-a01n02]
[2026-09-03T05:28:39.179Z] 
[2026-09-03T05:28:39.179Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:28:39.179Z] ] ***
[2026-09-03T05:28:39.918Z] ok: [an-a01n02]
[2026-09-03T05:28:39.918Z] 
[2026-09-03T05:28:39.918Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:28:39.918Z] ] ***
[2026-09-03T05:28:55.980Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to respond to ssh (60 retries left).
[2026-09-03T05:28:55.980Z] ok: [an-a01n02]
[2026-09-03T05:28:55.980Z] 
[2026-09-03T05:28:55.980Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:28:55.980Z] ] ***
[2026-09-03T05:28:55.980Z] ok: [an-a01n02]
[2026-09-03T05:28:55.980Z] 
[2026-09-03T05:28:55.980Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:28:55.980Z] ] ***
[2026-09-03T05:28:55.980Z] ok: [an-a01n02]
[2026-09-03T05:28:55.980Z] 
[2026-09-03T05:28:55.980Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-03T05:28:56.323Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:28:56.323Z] 
[2026-09-03T05:28:56.323Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01 -> localhost]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02 -> localhost]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:32:20.654Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:32:20.654Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:32:20.654Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ******************
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:32:20.654Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:32:20.654Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:32:20.654Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:32:20.654Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:32:20.654Z]   echo "pcs:"
[2026-09-03T05:32:20.654Z]   echo "$pcsstatus"
[2026-09-03T05:32:20.654Z]   exit 1
[2026-09-03T05:32:20.654Z] fi
[2026-09-03T05:32:20.654Z] exit 0
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:32:20.654Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:32:20.654Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:32:20.654Z] ret=0
[2026-09-03T05:32:20.654Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:32:20.654Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:32:20.654Z]   echo "$pcsstatus"
[2026-09-03T05:32:20.654Z]   ret=1
[2026-09-03T05:32:20.654Z] fi
[2026-09-03T05:32:20.654Z] echo "$hypervisor"
[2026-09-03T05:32:20.654Z] exit $ret
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-03T05:32:20.654Z] ok: [an-a01n01]
[2026-09-03T05:32:20.654Z] ok: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T05:32:20.654Z] ] **********
[2026-09-03T05:32:20.654Z] skipping: [an-a01n01]
[2026-09-03T05:32:20.654Z] changed: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T05:32:20.654Z] skipping: [an-a01n01]
[2026-09-03T05:32:20.654Z] skipping: [an-a01n02]
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-03T05:32:20.654Z] skipping: [an-a01n02]
[2026-09-03T05:32:20.654Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01
[2026-09-03T05:32:20.654Z] 
[2026-09-03T05:32:20.654Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-03T05:32:20.654Z] ] ***
[2026-09-03T05:33:30.420Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left).
[2026-09-03T05:33:30.420Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left).
[2026-09-03T05:33:30.420Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left).
[2026-09-03T05:33:30.420Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left).
[2026-09-03T05:33:30.420Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left).
[2026-09-03T05:33:30.420Z] ok: [an-a01n01]
[2026-09-03T05:33:30.420Z] 
[2026-09-03T05:33:30.420Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:33:30.420Z] skipping: [an-a01n02]
[2026-09-03T05:33:30.420Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01
[2026-09-03T05:33:30.420Z] 
[2026-09-03T05:33:30.420Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:33:30.420Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:33:30.420Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:33:30.420Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:33:30.420Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:33:30.420Z]   echo "pcs:"
[2026-09-03T05:33:30.420Z]   echo "$pcsstatus"
[2026-09-03T05:33:30.420Z]   exit 1
[2026-09-03T05:33:30.420Z] fi
[2026-09-03T05:33:30.420Z] exit 0
[2026-09-03T05:33:30.420Z] ] ***
[2026-09-03T05:34:28.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-03T05:34:28.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-03T05:34:28.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-03T05:34:28.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-03T05:34:28.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-03T05:34:28.399Z] ok: [an-a01n01]
[2026-09-03T05:34:28.399Z] 
[2026-09-03T05:34:28.399Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:34:28.399Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:34:28.399Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:34:28.399Z] ret=0
[2026-09-03T05:34:28.399Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:34:28.399Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:34:28.399Z]   echo "$pcsstatus"
[2026-09-03T05:34:28.399Z]   ret=1
[2026-09-03T05:34:28.399Z] fi
[2026-09-03T05:34:28.399Z] echo "$hypervisor"
[2026-09-03T05:34:28.399Z] exit $ret
[2026-09-03T05:34:28.399Z] ] ***
[2026-09-03T05:34:29.421Z] ok: [an-a01n01]
[2026-09-03T05:34:29.421Z] 
[2026-09-03T05:34:29.421Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02] => (item=1) 
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02] => (item=2) 
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02] => (item=3) 
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02] => (item=4) 
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02] => (item=5) 
[2026-09-03T05:34:29.421Z] skipping: [an-a01n02]
[2026-09-03T05:34:29.421Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1)
[2026-09-03T05:34:29.421Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2)
[2026-09-03T05:34:29.421Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3)
[2026-09-03T05:34:29.421Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4)
[2026-09-03T05:34:29.421Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5)
[2026-09-03T05:34:29.421Z] 
[2026-09-03T05:34:29.421Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:34:29.421Z] ] ***
[2026-09-03T05:34:45.487Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-03T05:34:45.487Z] ok: [an-a01n01]
[2026-09-03T05:34:45.487Z] 
[2026-09-03T05:34:45.487Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:34:45.487Z] ] ***
[2026-09-03T05:34:45.487Z] ok: [an-a01n01]
[2026-09-03T05:34:45.487Z] 
[2026-09-03T05:34:45.487Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:34:45.487Z] ] ***
[2026-09-03T05:34:45.487Z] ok: [an-a01n01]
[2026-09-03T05:34:45.487Z] 
[2026-09-03T05:34:45.487Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:34:45.487Z] ] ***
[2026-09-03T05:34:46.222Z] ok: [an-a01n01]
[2026-09-03T05:34:46.222Z] 
[2026-09-03T05:34:46.222Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:34:46.222Z] ] ***
[2026-09-03T05:34:46.956Z] ok: [an-a01n01]
[2026-09-03T05:34:46.956Z] 
[2026-09-03T05:34:46.956Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-03T05:34:46.956Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:34:46.956Z] 
[2026-09-03T05:34:46.956Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T05:38:11.068Z] ok: [an-a01n02 -> localhost]
[2026-09-03T05:38:11.068Z] ok: [an-a01n01 -> localhost]
[2026-09-03T05:38:11.068Z] 
[2026-09-03T05:38:11.068Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-03T05:38:11.068Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:38:11.068Z] 
[2026-09-03T05:38:11.069Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:38:11.069Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:38:11.069Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:38:11.069Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ******************
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:38:11.069Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:38:11.069Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:38:11.069Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:38:11.069Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:38:11.069Z]   echo "pcs:"
[2026-09-03T05:38:11.069Z]   echo "$pcsstatus"
[2026-09-03T05:38:11.069Z]   exit 1
[2026-09-03T05:38:11.069Z] fi
[2026-09-03T05:38:11.069Z] exit 0
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:38:11.069Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:38:11.069Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:38:11.069Z] ret=0
[2026-09-03T05:38:11.069Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:38:11.069Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:38:11.069Z]   echo "$pcsstatus"
[2026-09-03T05:38:11.069Z]   ret=1
[2026-09-03T05:38:11.069Z] fi
[2026-09-03T05:38:11.069Z] echo "$hypervisor"
[2026-09-03T05:38:11.069Z] exit $ret
[2026-09-03T05:38:11.069Z] ] ***
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-03T05:38:11.069Z] ok: [an-a01n01]
[2026-09-03T05:38:11.069Z] ok: [an-a01n02]
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-03T05:38:11.069Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:38:11.069Z] 
[2026-09-03T05:38:11.069Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-03T05:38:11.069Z] ] **********
[2026-09-03T05:38:11.410Z] skipping: [an-a01n02]
[2026-09-03T05:38:11.410Z] changed: [an-a01n01]
[2026-09-03T05:38:11.410Z] 
[2026-09-03T05:38:11.410Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-03T05:38:11.410Z] skipping: [an-a01n01]
[2026-09-03T05:38:11.410Z] skipping: [an-a01n02]
[2026-09-03T05:38:11.410Z] 
[2026-09-03T05:38:11.410Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-03T05:38:11.410Z] skipping: [an-a01n01]
[2026-09-03T05:38:11.410Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-03T05:38:11.410Z] 
[2026-09-03T05:38:11.410Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-03T05:38:11.410Z] ] ***
[2026-09-03T05:39:20.969Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-03T05:39:20.969Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-03T05:39:20.969Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-03T05:39:20.969Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-03T05:39:20.969Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-03T05:39:20.969Z] ok: [an-a01n02]
[2026-09-03T05:39:20.969Z] 
[2026-09-03T05:39:20.969Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-03T05:39:20.969Z] skipping: [an-a01n01]
[2026-09-03T05:39:20.969Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-03T05:39:20.969Z] 
[2026-09-03T05:39:20.969Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:39:20.969Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:39:20.969Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:39:20.969Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:39:20.969Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:39:20.969Z]   echo "pcs:"
[2026-09-03T05:39:20.969Z]   echo "$pcsstatus"
[2026-09-03T05:39:20.969Z]   exit 1
[2026-09-03T05:39:20.969Z] fi
[2026-09-03T05:39:20.969Z] exit 0
[2026-09-03T05:39:20.969Z] ] ***
[2026-09-03T05:40:19.015Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-03T05:40:19.015Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-03T05:40:19.015Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-03T05:40:19.015Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-03T05:40:19.015Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-03T05:40:19.015Z] ok: [an-a01n02]
[2026-09-03T05:40:19.015Z] 
[2026-09-03T05:40:19.015Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:40:19.015Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:40:19.015Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:40:19.015Z] ret=0
[2026-09-03T05:40:19.015Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:40:19.015Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:40:19.015Z]   echo "$pcsstatus"
[2026-09-03T05:40:19.015Z]   ret=1
[2026-09-03T05:40:19.015Z] fi
[2026-09-03T05:40:19.015Z] echo "$hypervisor"
[2026-09-03T05:40:19.015Z] exit $ret
[2026-09-03T05:40:19.015Z] ] ***
[2026-09-03T05:40:19.356Z] ok: [an-a01n02]
[2026-09-03T05:40:19.356Z] 
[2026-09-03T05:40:19.356Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01] => (item=1) 
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01] => (item=2) 
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01] => (item=3) 
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01] => (item=4) 
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01] => (item=5) 
[2026-09-03T05:40:19.356Z] skipping: [an-a01n01]
[2026-09-03T05:40:19.357Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-03T05:40:19.357Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-03T05:40:19.357Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-03T05:40:19.357Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-03T05:40:19.357Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-03T05:40:19.357Z] 
[2026-09-03T05:40:19.357Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:40:19.357Z] ] ***
[2026-09-03T05:40:35.579Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-03T05:40:35.579Z] ok: [an-a01n02]
[2026-09-03T05:40:35.579Z] 
[2026-09-03T05:40:35.579Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:40:35.579Z] ] ***
[2026-09-03T05:40:35.579Z] ok: [an-a01n02]
[2026-09-03T05:40:35.579Z] 
[2026-09-03T05:40:35.579Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:40:35.579Z] ] ***
[2026-09-03T05:40:35.579Z] ok: [an-a01n02]
[2026-09-03T05:40:35.579Z] 
[2026-09-03T05:40:35.579Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:40:35.579Z] ] ***
[2026-09-03T05:40:36.314Z] ok: [an-a01n02]
[2026-09-03T05:40:36.314Z] 
[2026-09-03T05:40:36.314Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T05:40:36.314Z] ] ***
[2026-09-03T05:40:37.048Z] ok: [an-a01n02]
[2026-09-03T05:40:37.048Z] 
[2026-09-03T05:40:37.048Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-03T05:40:37.048Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-03T05:40:37.048Z] 
[2026-09-03T05:40:37.048Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n02 -> localhost]
[2026-09-03T05:44:01.817Z] ok: [an-a01n01 -> localhost]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:44:01.817Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:44:01.817Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:44:01.817Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] PLAY [basic server tests] ******************************************************
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T05:44:01.817Z] ok: [an-a01n01]
[2026-09-03T05:44:01.817Z] ok: [an-a01n02]
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T05:44:01.817Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:01.817Z] 
[2026-09-03T05:44:01.817Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:44:01.817Z] ] ***
[2026-09-03T05:44:02.553Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:44:02.553Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:44:02.553Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:44:02.553Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:44:02.553Z] 
[2026-09-03T05:44:02.553Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-03T05:44:02.553Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:02.553Z] 
[2026-09-03T05:44:02.553Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-03T05:44:02.553Z] if [ -z "$storage" ]; then
[2026-09-03T05:44:02.553Z]   echo "Unable to detect shared storage"
[2026-09-03T05:44:02.553Z]   exit 1
[2026-09-03T05:44:02.553Z] fi
[2026-09-03T05:44:02.553Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T05:44:02.553Z] ] ***
[2026-09-03T05:44:08.575Z] changed: [an-a01n01]
[2026-09-03T05:44:08.575Z] changed: [an-a01n02]
[2026-09-03T05:44:08.575Z] 
[2026-09-03T05:44:08.575Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-03T05:44:08.575Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:08.575Z] 
[2026-09-03T05:44:08.575Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-03T05:44:08.575Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-03T05:44:08.575Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-03T05:44:08.575Z]   echo "Unable to detect all deploy iso"
[2026-09-03T05:44:08.575Z]   echo "Detected isos"
[2026-09-03T05:44:08.575Z]   echo "$isos"
[2026-09-03T05:44:08.575Z]   exit 1
[2026-09-03T05:44:08.575Z] fi
[2026-09-03T05:44:08.575Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-03T05:44:08.575Z] if [ -z "$installiso" ]; then
[2026-09-03T05:44:08.575Z]   echo "Unable to detect install iso"
[2026-09-03T05:44:08.575Z]   echo "Detected isos"
[2026-09-03T05:44:08.575Z]   echo "$isos"
[2026-09-03T05:44:08.575Z]   exit 1
[2026-09-03T05:44:08.575Z] fi
[2026-09-03T05:44:08.575Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T05:44:08.575Z] ] ***
[2026-09-03T05:44:14.604Z] changed: [an-a01n01]
[2026-09-03T05:44:14.604Z] changed: [an-a01n02]
[2026-09-03T05:44:14.604Z] 
[2026-09-03T05:44:14.604Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-03T05:44:14.604Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:14.604Z] 
[2026-09-03T05:44:14.604Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:44:14.604Z] ] ***
[2026-09-03T05:44:15.340Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.340Z] ok: [an-a01n01]
[2026-09-03T05:44:15.340Z] 
[2026-09-03T05:44:15.340Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T05:44:15.340Z]   --ci-test \
[2026-09-03T05:44:15.340Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T05:44:15.340Z]   --os almalinux9 \
[2026-09-03T05:44:15.340Z]   --cpu 2 --ram 4G \
[2026-09-03T05:44:15.340Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T05:44:15.340Z]   --install-media {{ installisouuid }} \
[2026-09-03T05:44:15.340Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T05:44:15.340Z] ] ***
[2026-09-03T05:44:15.340Z] skipping: [an-a01n01]
[2026-09-03T05:44:15.340Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.340Z] 
[2026-09-03T05:44:15.340Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T05:44:15.682Z] skipping: [an-a01n01]
[2026-09-03T05:44:15.682Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.682Z] 
[2026-09-03T05:44:15.682Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:44:15.682Z] skipping: [an-a01n01]
[2026-09-03T05:44:15.682Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.682Z] 
[2026-09-03T05:44:15.682Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T05:44:15.682Z] skipping: [an-a01n01]
[2026-09-03T05:44:15.682Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.682Z] 
[2026-09-03T05:44:15.682Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T05:44:15.682Z] skipping: [an-a01n01]
[2026-09-03T05:44:15.682Z] skipping: [an-a01n02]
[2026-09-03T05:44:15.682Z] 
[2026-09-03T05:44:15.682Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] ***
[2026-09-03T05:44:16.024Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:16.024Z] 
[2026-09-03T05:44:16.024Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:44:16.024Z] ] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] ok: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T05:44:17.735Z]   --ci-test \
[2026-09-03T05:44:17.735Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T05:44:17.735Z]   --os almalinux9 \
[2026-09-03T05:44:17.735Z]   --cpu 2 --ram 4G \
[2026-09-03T05:44:17.735Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T05:44:17.735Z]   --install-media {{ installisouuid }} \
[2026-09-03T05:44:17.735Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T05:44:17.735Z] ] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] skipping: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] skipping: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] skipping: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] skipping: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T05:44:17.735Z] skipping: [an-a01n01]
[2026-09-03T05:44:17.735Z] skipping: [an-a01n02]
[2026-09-03T05:44:17.735Z] 
[2026-09-03T05:44:17.735Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-03T05:44:18.076Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:18.076Z] 
[2026-09-03T05:44:18.076Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:44:18.076Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:44:18.076Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:44:18.076Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:44:18.076Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:44:18.076Z]   echo "pcs:"
[2026-09-03T05:44:18.076Z]   echo "$pcsstatus"
[2026-09-03T05:44:18.076Z]   exit 1
[2026-09-03T05:44:18.076Z] fi
[2026-09-03T05:44:18.076Z] exit 0
[2026-09-03T05:44:18.076Z] ] ***
[2026-09-03T05:44:20.400Z] ok: [an-a01n01]
[2026-09-03T05:44:20.400Z] ok: [an-a01n02]
[2026-09-03T05:44:20.400Z] 
[2026-09-03T05:44:20.400Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:44:20.400Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:44:20.400Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:44:20.400Z] ret=0
[2026-09-03T05:44:20.400Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:44:20.400Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:44:20.400Z]   echo "$pcsstatus"
[2026-09-03T05:44:20.400Z]   ret=1
[2026-09-03T05:44:20.400Z] fi
[2026-09-03T05:44:20.400Z] echo "$hypervisor"
[2026-09-03T05:44:20.400Z] exit $ret
[2026-09-03T05:44:20.400Z] ] ***
[2026-09-03T05:44:22.721Z] ok: [an-a01n01]
[2026-09-03T05:44:22.721Z] ok: [an-a01n02]
[2026-09-03T05:44:22.721Z] 
[2026-09-03T05:44:22.721Z] PLAY [delete extra server tests] ***********************************************
[2026-09-03T05:44:22.721Z] 
[2026-09-03T05:44:22.721Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T05:44:24.431Z] ok: [an-a01n01]
[2026-09-03T05:44:24.431Z] ok: [an-a01n02]
[2026-09-03T05:44:24.431Z] 
[2026-09-03T05:44:24.431Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T05:44:24.431Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:44:24.431Z] 
[2026-09-03T05:44:24.431Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:44:24.431Z] ] ***
[2026-09-03T05:44:29.295Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:44:29.295Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:44:29.295Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:44:29.295Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:44:29.295Z] 
[2026-09-03T05:44:29.295Z] TASK [delete servers _raw_params=../../common/delete_server.yml] ***************
[2026-09-03T05:44:29.295Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T05:44:29.295Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T05:44:29.295Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T05:44:29.295Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T05:44:29.295Z] 
[2026-09-03T05:44:29.295Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:44:29.295Z] ] ***
[2026-09-03T05:44:30.481Z] skipping: [an-a01n01]
[2026-09-03T05:44:30.481Z] ok: [an-a01n02]
[2026-09-03T05:44:30.481Z] 
[2026-09-03T05:44:30.481Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T05:44:30.481Z]   --force \
[2026-09-03T05:44:30.481Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T05:44:30.481Z] ] ***
[2026-09-03T05:44:41.462Z] skipping: [an-a01n01]
[2026-09-03T05:44:41.462Z] changed: [an-a01n02]
[2026-09-03T05:44:41.462Z] 
[2026-09-03T05:44:41.462Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:44:41.462Z] ] ***
[2026-09-03T05:45:09.693Z] skipping: [an-a01n01]
[2026-09-03T05:45:09.693Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left).
[2026-09-03T05:45:09.693Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left).
[2026-09-03T05:45:09.693Z] ok: [an-a01n02]
[2026-09-03T05:45:09.693Z] 
[2026-09-03T05:45:09.693Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:45:09.693Z] ] ***
[2026-09-03T05:45:09.693Z] skipping: [an-a01n01]
[2026-09-03T05:45:09.693Z] ok: [an-a01n02]
[2026-09-03T05:45:09.693Z] 
[2026-09-03T05:45:09.693Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T05:45:09.693Z] ] ***
[2026-09-03T05:45:09.693Z] skipping: [an-a01n01]
[2026-09-03T05:45:09.693Z] ok: [an-a01n02]
[2026-09-03T05:45:09.693Z] 
[2026-09-03T05:45:09.693Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:45:09.693Z] ] ***
[2026-09-03T05:45:09.693Z] skipping: [an-a01n01]
[2026-09-03T05:45:09.693Z] ok: [an-a01n02]
[2026-09-03T05:45:09.693Z] 
[2026-09-03T05:45:09.693Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T05:45:09.693Z]   --force \
[2026-09-03T05:45:09.693Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T05:45:09.693Z] ] ***
[2026-09-03T05:45:17.093Z] skipping: [an-a01n01]
[2026-09-03T05:45:17.093Z] changed: [an-a01n02]
[2026-09-03T05:45:17.093Z] 
[2026-09-03T05:45:17.093Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:45:17.093Z] ] ***
[2026-09-03T05:46:40.648Z] skipping: [an-a01n01]
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left).
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left).
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left).
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left).
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left).
[2026-09-03T05:46:40.648Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left).
[2026-09-03T05:46:40.648Z] ok: [an-a01n02]
[2026-09-03T05:46:40.648Z] 
[2026-09-03T05:46:40.648Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:46:40.648Z] ] ***
[2026-09-03T05:46:40.648Z] skipping: [an-a01n01]
[2026-09-03T05:46:40.648Z] ok: [an-a01n02]
[2026-09-03T05:46:40.648Z] 
[2026-09-03T05:46:40.648Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T05:46:40.648Z] ] ***
[2026-09-03T05:46:40.648Z] skipping: [an-a01n01]
[2026-09-03T05:46:40.648Z] ok: [an-a01n02]
[2026-09-03T05:46:40.648Z] 
[2026-09-03T05:46:40.648Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:46:40.648Z] ] ***
[2026-09-03T05:46:40.648Z] skipping: [an-a01n01]
[2026-09-03T05:46:40.648Z] ok: [an-a01n02]
[2026-09-03T05:46:40.648Z] 
[2026-09-03T05:46:40.648Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T05:46:40.648Z]   --force \
[2026-09-03T05:46:40.648Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T05:46:40.648Z] ] ***
[2026-09-03T05:46:40.648Z] skipping: [an-a01n01]
[2026-09-03T05:46:40.648Z] changed: [an-a01n02]
[2026-09-03T05:46:40.648Z] 
[2026-09-03T05:46:40.648Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:46:40.648Z] ] ***
[2026-09-03T05:48:04.011Z] skipping: [an-a01n01]
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (55 retries left).
[2026-09-03T05:48:04.011Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (54 retries left).
[2026-09-03T05:48:04.011Z] ok: [an-a01n02]
[2026-09-03T05:48:04.011Z] 
[2026-09-03T05:48:04.011Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:48:04.011Z] ] ***
[2026-09-03T05:48:04.011Z] skipping: [an-a01n01]
[2026-09-03T05:48:04.011Z] ok: [an-a01n02]
[2026-09-03T05:48:04.011Z] 
[2026-09-03T05:48:04.011Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T05:48:04.011Z] ] ***
[2026-09-03T05:48:04.011Z] skipping: [an-a01n01]
[2026-09-03T05:48:04.011Z] ok: [an-a01n02]
[2026-09-03T05:48:04.011Z] 
[2026-09-03T05:48:04.011Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:48:04.011Z] ] ***
[2026-09-03T05:48:05.718Z] skipping: [an-a01n01]
[2026-09-03T05:48:05.718Z] ok: [an-a01n02]
[2026-09-03T05:48:05.718Z] 
[2026-09-03T05:48:05.718Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T05:48:05.718Z]   --force \
[2026-09-03T05:48:05.718Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T05:48:05.718Z] ] ***
[2026-09-03T05:48:14.754Z] skipping: [an-a01n01]
[2026-09-03T05:48:14.754Z] changed: [an-a01n02]
[2026-09-03T05:48:14.754Z] 
[2026-09-03T05:48:14.754Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:48:14.754Z] ] ***
[2026-09-03T05:49:24.289Z] skipping: [an-a01n01]
[2026-09-03T05:49:24.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left).
[2026-09-03T05:49:24.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left).
[2026-09-03T05:49:24.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left).
[2026-09-03T05:49:24.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left).
[2026-09-03T05:49:24.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left).
[2026-09-03T05:49:24.289Z] ok: [an-a01n02]
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:49:24.289Z] ] ***
[2026-09-03T05:49:24.289Z] skipping: [an-a01n01]
[2026-09-03T05:49:24.289Z] ok: [an-a01n02]
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T05:49:24.289Z] ] ***
[2026-09-03T05:49:24.289Z] skipping: [an-a01n01]
[2026-09-03T05:49:24.289Z] ok: [an-a01n02]
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] PLAY [create server for DR testing] ********************************************
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T05:49:24.289Z] ok: [an-a01n02]
[2026-09-03T05:49:24.289Z] ok: [an-a01n01]
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T05:49:24.289Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T05:49:24.289Z] ] ***
[2026-09-03T05:49:24.289Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T05:49:24.289Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T05:49:24.289Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T05:49:24.289Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-03T05:49:24.289Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-03T05:49:24.289Z] 
[2026-09-03T05:49:24.289Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-03T05:49:24.289Z] if [ -z "$storage" ]; then
[2026-09-03T05:49:24.289Z]   echo "Unable to detect shared storage"
[2026-09-03T05:49:24.289Z]   exit 1
[2026-09-03T05:49:24.289Z] fi
[2026-09-03T05:49:24.289Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T05:49:24.289Z] ] ***
[2026-09-03T05:49:27.324Z] changed: [an-a01n02]
[2026-09-03T05:49:27.324Z] changed: [an-a01n01]
[2026-09-03T05:49:27.324Z] 
[2026-09-03T05:49:27.324Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-03T05:49:27.324Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-03T05:49:27.324Z] 
[2026-09-03T05:49:27.324Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-03T05:49:27.324Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-03T05:49:27.324Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-03T05:49:27.324Z]   echo "Unable to detect all deploy iso"
[2026-09-03T05:49:27.324Z]   echo "Detected isos"
[2026-09-03T05:49:27.324Z]   echo "$isos"
[2026-09-03T05:49:27.324Z]   exit 1
[2026-09-03T05:49:27.324Z] fi
[2026-09-03T05:49:27.324Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-03T05:49:27.324Z] if [ -z "$installiso" ]; then
[2026-09-03T05:49:27.324Z]   echo "Unable to detect install iso"
[2026-09-03T05:49:27.324Z]   echo "Detected isos"
[2026-09-03T05:49:27.324Z]   echo "$isos"
[2026-09-03T05:49:27.324Z]   exit 1
[2026-09-03T05:49:27.324Z] fi
[2026-09-03T05:49:27.324Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T05:49:27.324Z] ] ***
[2026-09-03T05:49:32.182Z] changed: [an-a01n01]
[2026-09-03T05:49:32.182Z] changed: [an-a01n02]
[2026-09-03T05:49:32.182Z] 
[2026-09-03T05:49:32.182Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-03T05:49:32.182Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-03T05:49:32.182Z] 
[2026-09-03T05:49:32.182Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T05:49:32.182Z] ] ***
[2026-09-03T05:49:33.896Z] skipping: [an-a01n02]
[2026-09-03T05:49:33.896Z] ok: [an-a01n01]
[2026-09-03T05:49:33.896Z] 
[2026-09-03T05:49:33.896Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T05:49:33.896Z]   --ci-test \
[2026-09-03T05:49:33.896Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T05:49:33.896Z]   --os almalinux9 \
[2026-09-03T05:49:33.896Z]   --cpu 2 --ram 4G \
[2026-09-03T05:49:33.896Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T05:49:33.896Z]   --install-media {{ installisouuid }} \
[2026-09-03T05:49:33.896Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T05:49:33.896Z] ] ***
[2026-09-03T05:49:33.896Z] skipping: [an-a01n01]
[2026-09-03T05:49:33.896Z] skipping: [an-a01n02]
[2026-09-03T05:49:33.896Z] 
[2026-09-03T05:49:33.896Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T05:49:33.896Z] skipping: [an-a01n01]
[2026-09-03T05:49:33.896Z] skipping: [an-a01n02]
[2026-09-03T05:49:33.896Z] 
[2026-09-03T05:49:33.896Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T05:49:33.896Z] skipping: [an-a01n01]
[2026-09-03T05:49:33.896Z] skipping: [an-a01n02]
[2026-09-03T05:49:33.896Z] 
[2026-09-03T05:49:33.896Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T05:49:34.238Z] skipping: [an-a01n01]
[2026-09-03T05:49:34.238Z] skipping: [an-a01n02]
[2026-09-03T05:49:34.238Z] 
[2026-09-03T05:49:34.239Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T05:49:34.239Z] skipping: [an-a01n01]
[2026-09-03T05:49:34.239Z] skipping: [an-a01n02]
[2026-09-03T05:49:34.239Z] 
[2026-09-03T05:49:34.239Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-03T05:49:34.239Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-03T05:49:34.239Z] 
[2026-09-03T05:49:34.239Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:49:34.239Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T05:49:34.239Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T05:49:34.239Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T05:49:34.239Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T05:49:34.239Z]   echo "pcs:"
[2026-09-03T05:49:34.239Z]   echo "$pcsstatus"
[2026-09-03T05:49:34.239Z]   exit 1
[2026-09-03T05:49:34.239Z] fi
[2026-09-03T05:49:34.239Z] exit 0
[2026-09-03T05:49:34.239Z] ] ***
[2026-09-03T05:49:36.561Z] ok: [an-a01n01]
[2026-09-03T05:49:36.561Z] ok: [an-a01n02]
[2026-09-03T05:49:36.561Z] 
[2026-09-03T05:49:36.561Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T05:49:36.561Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T05:49:36.561Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T05:49:36.561Z] ret=0
[2026-09-03T05:49:36.561Z] if [ "$servers"  != "1" ]; then
[2026-09-03T05:49:36.561Z]   echo "Not all servers are running on the same node!"
[2026-09-03T05:49:36.561Z]   echo "$pcsstatus"
[2026-09-03T05:49:36.561Z]   ret=1
[2026-09-03T05:49:36.561Z] fi
[2026-09-03T05:49:36.561Z] echo "$hypervisor"
[2026-09-03T05:49:36.561Z] exit $ret
[2026-09-03T05:49:36.561Z] ] ***
[2026-09-03T05:49:38.272Z] ok: [an-a01n01]
[2026-09-03T05:49:38.273Z] ok: [an-a01n02]
[2026-09-03T05:49:38.273Z] 
[2026-09-03T05:49:38.273Z] PLAY [DR tests] ****************************************************************
[2026-09-03T05:49:38.273Z] 
[2026-09-03T05:49:38.273Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T05:49:40.727Z] ok: [an-a01dr01]
[2026-09-03T05:49:40.727Z] 
[2026-09-03T05:49:40.727Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured
[2026-09-03T05:49:40.727Z] ] ***
[2026-09-03T05:49:44.603Z] changed: [an-a01dr01]
[2026-09-03T05:49:44.603Z] 
[2026-09-03T05:49:44.603Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g')
[2026-09-03T05:49:44.603Z] echo "DB registered hostname: $drhostname"
[2026-09-03T05:49:44.603Z] echo "current hostname: $(hostname)"
[2026-09-03T05:49:44.603Z] if [ "$drhostname" != "$(hostname)" ]; then
[2026-09-03T05:49:44.603Z]   echo "DR hostnames are not matching"
[2026-09-03T05:49:44.603Z]   exit 1
[2026-09-03T05:49:44.603Z] fi
[2026-09-03T05:49:44.603Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then
[2026-09-03T05:49:44.603Z]   echo "DR hostnames are not matching drnodes an-a01dr01"
[2026-09-03T05:49:44.603Z]   exit 1
[2026-09-03T05:49:44.603Z] fi
[2026-09-03T05:49:44.603Z] ] ***
[2026-09-03T05:49:49.465Z] changed: [an-a01dr01]
[2026-09-03T05:49:49.465Z] 
[2026-09-03T05:49:49.465Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show  | grep -q "No linked DR hosts"; then
[2026-09-03T05:49:49.465Z]   echo "Detected a linked DR host!"
[2026-09-03T05:49:49.465Z]   exit 1
[2026-09-03T05:49:49.465Z] fi
[2026-09-03T05:49:49.465Z] ] ***
[2026-09-03T05:49:53.344Z] changed: [an-a01dr01]
[2026-09-03T05:49:53.344Z] 
[2026-09-03T05:49:53.344Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01
[2026-09-03T05:49:53.344Z] ] ***
[2026-09-03T05:49:58.201Z] changed: [an-a01dr01]
[2026-09-03T05:49:58.201Z] 
[2026-09-03T05:49:58.201Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then
[2026-09-03T05:49:58.201Z]   echo "DR host failed to link to anvil"
[2026-09-03T05:49:58.201Z]   exit 1
[2026-09-03T05:49:58.201Z] fi
[2026-09-03T05:49:58.201Z] ] ***
[2026-09-03T05:50:03.198Z] changed: [an-a01dr01]
[2026-09-03T05:50:03.198Z] 
[2026-09-03T05:50:03.198Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##')
[2026-09-03T05:50:03.198Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}')
[2026-09-03T05:50:03.198Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then
[2026-09-03T05:50:03.198Z]   echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]"
[2026-09-03T05:50:03.198Z]   exit 1
[2026-09-03T05:50:03.198Z] fi
[2026-09-03T05:50:03.198Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then
[2026-09-03T05:50:03.198Z]   echo "Unable to link local vg uuid to storage group"
[2026-09-03T05:50:03.198Z]   exit 1
[2026-09-03T05:50:03.198Z] fi
[2026-09-03T05:50:03.198Z] ] ***
[2026-09-03T05:50:09.222Z] changed: [an-a01dr01]
[2026-09-03T05:50:09.222Z] 
[2026-09-03T05:50:09.222Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then
[2026-09-03T05:50:09.222Z]   echo "Local vg does not appear to be linked to storage group"
[2026-09-03T05:50:09.222Z]   exit 1
[2026-09-03T05:50:09.222Z] fi
[2026-09-03T05:50:09.222Z] ] ***
[2026-09-03T05:50:13.096Z] changed: [an-a01dr01]
[2026-09-03T05:50:13.096Z] 
[2026-09-03T05:50:13.096Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then
[2026-09-03T05:50:13.096Z]   echo "Unable to add protection to server an-test-deploy1"
[2026-09-03T05:50:13.096Z]   exit 1
[2026-09-03T05:50:13.096Z] fi
[2026-09-03T05:50:13.096Z] ] ***
[2026-09-03T05:50:17.953Z] changed: [an-a01dr01]
[2026-09-03T05:50:17.953Z] 
[2026-09-03T05:50:17.953Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] ***
[2026-09-03T05:50:18.295Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01
[2026-09-03T05:50:18.295Z] 
[2026-09-03T05:50:18.295Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-03T05:50:18.295Z]   exit 1
[2026-09-03T05:50:18.295Z] fi
[2026-09-03T05:50:18.295Z] ] ***
[2026-09-03T05:51:06.779Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left).
[2026-09-03T05:51:06.779Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left).
[2026-09-03T05:51:06.779Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left).
[2026-09-03T05:51:06.779Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left).
[2026-09-03T05:51:06.779Z] ok: [an-a01dr01]
[2026-09-03T05:51:06.779Z] 
[2026-09-03T05:51:06.780Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in
[2026-09-03T05:51:06.780Z]   short-throw)
[2026-09-03T05:51:06.780Z]     expected=2
[2026-09-03T05:51:06.780Z]     protocol=A
[2026-09-03T05:51:06.780Z]   ;;
[2026-09-03T05:51:06.780Z]   sync)
[2026-09-03T05:51:06.780Z]     expected=3
[2026-09-03T05:51:06.780Z]     protocol=C
[2026-09-03T05:51:06.780Z]   ;;
[2026-09-03T05:51:06.780Z]   *)
[2026-09-03T05:51:06.780Z]     echo "unknown DRBD protection method"
[2026-09-03T05:51:06.780Z]     exit 1
[2026-09-03T05:51:06.780Z]   ;;
[2026-09-03T05:51:06.780Z] esac
[2026-09-03T05:51:06.780Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then
[2026-09-03T05:51:06.780Z]   echo "Unable to confirm correct DRBD protocol"
[2026-09-03T05:51:06.780Z]   exit 1
[2026-09-03T05:51:06.780Z] fi
[2026-09-03T05:51:06.780Z] ] ***
[2026-09-03T05:51:06.780Z] changed: [an-a01dr01]
[2026-09-03T05:51:06.780Z] 
[2026-09-03T05:51:06.780Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:51:06.780Z] ] ***
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (107 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (106 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (105 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (104 retries left).
[2026-09-03T05:54:15.821Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (103 retries left).
[2026-09-03T05:54:15.821Z] ok: [an-a01dr01]
[2026-09-03T05:54:15.821Z] 
[2026-09-03T05:54:15.821Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] ***
[2026-09-03T05:54:15.821Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01
[2026-09-03T05:54:15.821Z] 
[2026-09-03T05:54:15.821Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T05:54:15.821Z] ] ***
[2026-09-03T05:54:16.554Z] ok: [an-a01dr01]
[2026-09-03T05:54:16.554Z] 
[2026-09-03T05:54:16.554Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then
[2026-09-03T05:54:16.554Z]   echo "Unable to disconnect server from DR replication"
[2026-09-03T05:54:16.555Z]   exit 1
[2026-09-03T05:54:16.555Z] fi
[2026-09-03T05:54:16.555Z] ] ***
[2026-09-03T05:54:22.577Z] changed: [an-a01dr01]
[2026-09-03T05:54:22.577Z] 
[2026-09-03T05:54:22.577Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:54:22.577Z] ] ***
[2026-09-03T05:54:45.961Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-03T05:54:45.961Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left).
[2026-09-03T05:54:45.961Z] ok: [an-a01dr01]
[2026-09-03T05:54:45.961Z] 
[2026-09-03T05:54:45.961Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock"
[2026-09-03T05:54:45.961Z] sleep 120
[2026-09-03T05:54:45.961Z] ] ***
[2026-09-03T05:56:54.629Z] changed: [an-a01dr01 -> localhost]
[2026-09-03T05:56:54.629Z] 
[2026-09-03T05:56:54.629Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then
[2026-09-03T05:56:54.629Z]   echo "Unable to update server DR replications"
[2026-09-03T05:56:54.629Z]   exit 1
[2026-09-03T05:56:54.629Z] fi
[2026-09-03T05:56:54.629Z] ] ***
[2026-09-03T05:56:55.363Z] changed: [an-a01dr01]
[2026-09-03T05:56:55.363Z] 
[2026-09-03T05:56:55.363Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:56:55.363Z] ] ***
[2026-09-03T05:57:18.669Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-03T05:57:18.669Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-03T05:57:18.669Z] ok: [an-a01dr01]
[2026-09-03T05:57:18.669Z] 
[2026-09-03T05:57:18.669Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:57:18.669Z] ] ***
[2026-09-03T05:57:29.646Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-03T05:57:29.646Z] ok: [an-a01dr01]
[2026-09-03T05:57:29.646Z] 
[2026-09-03T05:57:29.646Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then
[2026-09-03T05:57:29.646Z]   echo "Unable to (re)connect server from DR replication"
[2026-09-03T05:57:29.646Z]   exit 1
[2026-09-03T05:57:29.646Z] fi
[2026-09-03T05:57:29.646Z] ] ***
[2026-09-03T05:57:35.670Z] changed: [an-a01dr01]
[2026-09-03T05:57:35.670Z] 
[2026-09-03T05:57:35.670Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:57:35.670Z] ] ***
[2026-09-03T05:58:09.278Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-03T05:58:09.278Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-03T05:58:09.278Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left).
[2026-09-03T05:58:09.278Z] ok: [an-a01dr01]
[2026-09-03T05:58:09.278Z] 
[2026-09-03T05:58:09.278Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then
[2026-09-03T05:58:09.278Z]   echo "Unable to remove protection from server an-test-deploy1"
[2026-09-03T05:58:09.278Z]   exit 1
[2026-09-03T05:58:09.278Z] fi
[2026-09-03T05:58:09.278Z] ] ***
[2026-09-03T05:58:13.147Z] changed: [an-a01dr01]
[2026-09-03T05:58:13.147Z] 
[2026-09-03T05:58:13.147Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] ***
[2026-09-03T05:58:13.147Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01
[2026-09-03T05:58:13.147Z] 
[2026-09-03T05:58:13.147Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T05:58:13.147Z] ] ***
[2026-09-03T05:58:46.825Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-03T05:58:46.825Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left).
[2026-09-03T05:58:46.825Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left).
[2026-09-03T05:58:46.825Z] ok: [an-a01dr01]
[2026-09-03T05:58:46.825Z] 
[2026-09-03T05:58:46.825Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T05:58:46.825Z] ] ***
[2026-09-03T05:58:46.825Z] ok: [an-a01dr01]
[2026-09-03T05:58:46.825Z] 
[2026-09-03T05:58:46.825Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-03T05:58:46.825Z]   exit 1
[2026-09-03T05:58:46.825Z] fi
[2026-09-03T05:58:46.825Z] ] ***
[2026-09-03T05:59:20.402Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left).
[2026-09-03T05:59:20.402Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left).
[2026-09-03T05:59:20.402Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left).
[2026-09-03T05:59:20.402Z] ok: [an-a01dr01]
[2026-09-03T05:59:20.402Z] 
[2026-09-03T05:59:20.402Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then
[2026-09-03T05:59:20.402Z]   echo "drbdadm status still reports connection to DR node (#888)"
[2026-09-03T05:59:20.402Z]   #exit 1
[2026-09-03T05:59:20.402Z] fi
[2026-09-03T05:59:20.402Z] ] ***
[2026-09-03T05:59:20.402Z] changed: [an-a01dr01 -> an-a01n01]
[2026-09-03T05:59:20.402Z] 
[2026-09-03T05:59:20.402Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then
[2026-09-03T05:59:20.402Z]   echo "Unable to add protection to server an-test-deploy1"
[2026-09-03T05:59:20.402Z]   exit 1
[2026-09-03T05:59:20.402Z] fi
[2026-09-03T05:59:20.402Z] ] ***
[2026-09-03T05:59:25.254Z] changed: [an-a01dr01]
[2026-09-03T05:59:25.254Z] 
[2026-09-03T05:59:25.254Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] ***
[2026-09-03T05:59:25.254Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01
[2026-09-03T05:59:25.254Z] 
[2026-09-03T05:59:25.254Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-03T05:59:25.254Z]   exit 1
[2026-09-03T05:59:25.254Z] fi
[2026-09-03T05:59:25.254Z] ] ***
[2026-09-03T06:00:23.440Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left).
[2026-09-03T06:00:23.440Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left).
[2026-09-03T06:00:23.440Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left).
[2026-09-03T06:00:23.440Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left).
[2026-09-03T06:00:23.440Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (116 retries left).
[2026-09-03T06:00:23.440Z] ok: [an-a01dr01]
[2026-09-03T06:00:23.440Z] 
[2026-09-03T06:00:23.440Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in
[2026-09-03T06:00:23.440Z]   short-throw)
[2026-09-03T06:00:23.440Z]     expected=2
[2026-09-03T06:00:23.440Z]     protocol=A
[2026-09-03T06:00:23.440Z]   ;;
[2026-09-03T06:00:23.440Z]   sync)
[2026-09-03T06:00:23.440Z]     expected=3
[2026-09-03T06:00:23.440Z]     protocol=C
[2026-09-03T06:00:23.440Z]   ;;
[2026-09-03T06:00:23.440Z]   *)
[2026-09-03T06:00:23.440Z]     echo "unknown DRBD protection method"
[2026-09-03T06:00:23.440Z]     exit 1
[2026-09-03T06:00:23.440Z]   ;;
[2026-09-03T06:00:23.440Z] esac
[2026-09-03T06:00:23.440Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then
[2026-09-03T06:00:23.440Z]   echo "Unable to confirm correct DRBD protocol"
[2026-09-03T06:00:23.440Z]   exit 1
[2026-09-03T06:00:23.440Z] fi
[2026-09-03T06:00:23.440Z] ] ***
[2026-09-03T06:00:23.440Z] changed: [an-a01dr01]
[2026-09-03T06:00:23.440Z] 
[2026-09-03T06:00:23.440Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T06:00:23.440Z] ] ***
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left).
[2026-09-03T06:02:32.092Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left).
[2026-09-03T06:02:32.092Z] ok: [an-a01dr01]
[2026-09-03T06:02:32.092Z] 
[2026-09-03T06:02:32.092Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] ***
[2026-09-03T06:02:32.092Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01
[2026-09-03T06:02:32.092Z] 
[2026-09-03T06:02:32.092Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T06:02:32.092Z] ] ***
[2026-09-03T06:02:32.092Z] ok: [an-a01dr01]
[2026-09-03T06:02:32.092Z] 
[2026-09-03T06:02:32.092Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then
[2026-09-03T06:02:32.092Z]   echo "Unable to disconnect server from DR replication"
[2026-09-03T06:02:32.092Z]   exit 1
[2026-09-03T06:02:32.092Z] fi
[2026-09-03T06:02:32.092Z] ] ***
[2026-09-03T06:02:35.964Z] changed: [an-a01dr01]
[2026-09-03T06:02:35.964Z] 
[2026-09-03T06:02:35.964Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:02:35.964Z] ] ***
[2026-09-03T06:03:09.548Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-03T06:03:09.548Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left).
[2026-09-03T06:03:09.548Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (118 retries left).
[2026-09-03T06:03:09.548Z] ok: [an-a01dr01]
[2026-09-03T06:03:09.548Z] 
[2026-09-03T06:03:09.548Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock"
[2026-09-03T06:03:09.548Z] sleep 120
[2026-09-03T06:03:09.548Z] ] ***
[2026-09-03T06:05:33.395Z] changed: [an-a01dr01 -> localhost]
[2026-09-03T06:05:33.395Z] 
[2026-09-03T06:05:33.395Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then
[2026-09-03T06:05:33.395Z]   echo "Unable to update server DR replications"
[2026-09-03T06:05:33.395Z]   exit 1
[2026-09-03T06:05:33.395Z] fi
[2026-09-03T06:05:33.395Z] ] ***
[2026-09-03T06:05:33.395Z] changed: [an-a01dr01]
[2026-09-03T06:05:33.395Z] 
[2026-09-03T06:05:33.395Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:05:33.395Z] ] ***
[2026-09-03T06:05:49.451Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-03T06:05:49.451Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-03T06:05:49.451Z] ok: [an-a01dr01]
[2026-09-03T06:05:49.451Z] 
[2026-09-03T06:05:49.451Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:05:49.451Z] ] ***
[2026-09-03T06:06:00.434Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-03T06:06:00.434Z] ok: [an-a01dr01]
[2026-09-03T06:06:00.434Z] 
[2026-09-03T06:06:00.434Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then
[2026-09-03T06:06:00.435Z]   echo "Unable to (re)connect server from DR replication"
[2026-09-03T06:06:00.435Z]   exit 1
[2026-09-03T06:06:00.435Z] fi
[2026-09-03T06:06:00.435Z] ] ***
[2026-09-03T06:06:06.457Z] changed: [an-a01dr01]
[2026-09-03T06:06:06.457Z] 
[2026-09-03T06:06:06.457Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:06:06.457Z] ] ***
[2026-09-03T06:06:29.742Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-03T06:06:29.742Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-03T06:06:29.742Z] ok: [an-a01dr01]
[2026-09-03T06:06:29.742Z] 
[2026-09-03T06:06:29.742Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then
[2026-09-03T06:06:29.742Z]   echo "Unable to remove protection from server an-test-deploy1"
[2026-09-03T06:06:29.742Z]   exit 1
[2026-09-03T06:06:29.742Z] fi
[2026-09-03T06:06:29.742Z] ] ***
[2026-09-03T06:06:33.616Z] changed: [an-a01dr01]
[2026-09-03T06:06:33.616Z] 
[2026-09-03T06:06:33.616Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] ***
[2026-09-03T06:06:33.616Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01
[2026-09-03T06:06:33.616Z] 
[2026-09-03T06:06:33.616Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:06:33.616Z] ] ***
[2026-09-03T06:07:21.990Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-03T06:07:21.990Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left).
[2026-09-03T06:07:21.990Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left).
[2026-09-03T06:07:21.990Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (57 retries left).
[2026-09-03T06:07:21.990Z] ok: [an-a01dr01]
[2026-09-03T06:07:21.990Z] 
[2026-09-03T06:07:21.990Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:07:21.990Z] ] ***
[2026-09-03T06:07:28.111Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed (60 retries left).
[2026-09-03T06:07:28.111Z] ok: [an-a01dr01]
[2026-09-03T06:07:28.111Z] 
[2026-09-03T06:07:28.111Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-03T06:07:28.111Z]   exit 1
[2026-09-03T06:07:28.111Z] fi
[2026-09-03T06:07:28.111Z] ] ***
[2026-09-03T06:08:01.696Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left).
[2026-09-03T06:08:01.696Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left).
[2026-09-03T06:08:01.696Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left).
[2026-09-03T06:08:01.696Z] ok: [an-a01dr01]
[2026-09-03T06:08:01.696Z] 
[2026-09-03T06:08:01.696Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then
[2026-09-03T06:08:01.696Z]   echo "drbdadm status still reports connection to DR node (#888)"
[2026-09-03T06:08:01.696Z]   #exit 1
[2026-09-03T06:08:01.696Z] fi
[2026-09-03T06:08:01.696Z] ] ***
[2026-09-03T06:08:01.696Z] changed: [an-a01dr01 -> an-a01n01]
[2026-09-03T06:08:01.696Z] 
[2026-09-03T06:08:01.696Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##')
[2026-09-03T06:08:01.696Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}')
[2026-09-03T06:08:01.696Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then
[2026-09-03T06:08:01.696Z]   echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]"
[2026-09-03T06:08:01.696Z]   exit 1
[2026-09-03T06:08:01.696Z] fi
[2026-09-03T06:08:01.696Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then
[2026-09-03T06:08:01.696Z]   echo "Unable to unlink local vg uuid to storage group"
[2026-09-03T06:08:01.696Z]   exit 1
[2026-09-03T06:08:01.696Z] fi
[2026-09-03T06:08:01.696Z] ] ***
[2026-09-03T06:08:07.720Z] changed: [an-a01dr01]
[2026-09-03T06:08:07.720Z] 
[2026-09-03T06:08:07.720Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then
[2026-09-03T06:08:07.720Z]   echo "Local vg does not appear to be unlinked from storage group (#886)"
[2026-09-03T06:08:07.720Z]   exit 1
[2026-09-03T06:08:07.720Z] fi
[2026-09-03T06:08:07.720Z] ] ***
[2026-09-03T06:08:11.686Z] changed: [an-a01dr01]
[2026-09-03T06:08:11.686Z] 
[2026-09-03T06:08:11.686Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01
[2026-09-03T06:08:11.686Z] ] ***
[2026-09-03T06:08:16.545Z] changed: [an-a01dr01]
[2026-09-03T06:08:16.545Z] 
[2026-09-03T06:08:16.545Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr  -vv --log-secure --show  | grep -q "No linked DR hosts"; then
[2026-09-03T06:08:16.545Z]   echo "Detected a linked DR host!"
[2026-09-03T06:08:16.545Z]   exit 1
[2026-09-03T06:08:16.545Z] fi
[2026-09-03T06:08:16.545Z] ] ***
[2026-09-03T06:08:21.403Z] changed: [an-a01dr01]
[2026-09-03T06:08:21.403Z] 
[2026-09-03T06:08:21.403Z] PLAY [upgrade server test] *****************************************************
[2026-09-03T06:08:21.403Z] 
[2026-09-03T06:08:21.403Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:08:23.115Z] ok: [an-a01n01]
[2026-09-03T06:08:23.115Z] ok: [an-a01n02]
[2026-09-03T06:08:23.115Z] 
[2026-09-03T06:08:23.115Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T06:08:23.115Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T06:08:23.115Z] 
[2026-09-03T06:08:23.115Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T06:08:23.115Z] ] ***
[2026-09-03T06:08:26.986Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T06:08:26.986Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T06:08:26.986Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T06:08:26.986Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T06:08:26.986Z] 
[2026-09-03T06:08:26.986Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-03T06:08:26.986Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-03T06:08:26.986Z] 
[2026-09-03T06:08:26.986Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-03T06:08:26.986Z] if [ -z "$storage" ]; then
[2026-09-03T06:08:26.986Z]   echo "Unable to detect shared storage"
[2026-09-03T06:08:26.986Z]   exit 1
[2026-09-03T06:08:26.986Z] fi
[2026-09-03T06:08:26.986Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T06:08:26.986Z] ] ***
[2026-09-03T06:08:33.030Z] changed: [an-a01n02]
[2026-09-03T06:08:33.030Z] changed: [an-a01n01]
[2026-09-03T06:08:33.030Z] 
[2026-09-03T06:08:33.030Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-03T06:08:33.030Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-03T06:08:33.030Z] 
[2026-09-03T06:08:33.030Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-03T06:08:33.030Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-03T06:08:33.030Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-03T06:08:33.030Z]   echo "Unable to detect all deploy iso"
[2026-09-03T06:08:33.030Z]   echo "Detected isos"
[2026-09-03T06:08:33.030Z]   echo "$isos"
[2026-09-03T06:08:33.030Z]   exit 1
[2026-09-03T06:08:33.030Z] fi
[2026-09-03T06:08:33.030Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-03T06:08:33.030Z] if [ -z "$installiso" ]; then
[2026-09-03T06:08:33.030Z]   echo "Unable to detect install iso"
[2026-09-03T06:08:33.030Z]   echo "Detected isos"
[2026-09-03T06:08:33.030Z]   echo "$isos"
[2026-09-03T06:08:33.030Z]   exit 1
[2026-09-03T06:08:33.030Z] fi
[2026-09-03T06:08:33.030Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-03T06:08:33.030Z] ] ***
[2026-09-03T06:08:37.892Z] changed: [an-a01n01]
[2026-09-03T06:08:37.892Z] changed: [an-a01n02]
[2026-09-03T06:08:37.892Z] 
[2026-09-03T06:08:37.892Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-03T06:08:37.892Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-03T06:08:37.892Z] 
[2026-09-03T06:08:37.892Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:08:37.892Z] ] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] ok: [an-a01n01]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 7c2810eb-4ff8-47c7-8fb5-c625e1c54de7 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-03T06:08:39.605Z]   --ci-test \
[2026-09-03T06:08:39.605Z]   --name an-test-deploy{{ servernum }} \
[2026-09-03T06:08:39.605Z]   --os almalinux9 \
[2026-09-03T06:08:39.605Z]   --cpu 2 --ram 4G \
[2026-09-03T06:08:39.605Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-03T06:08:39.605Z]   --install-media {{ installisouuid }} \
[2026-09-03T06:08:39.605Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-03T06:08:39.605Z] ] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n01]
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n01]
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n01]
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n01]
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-03T06:08:39.605Z] skipping: [an-a01n01]
[2026-09-03T06:08:39.605Z] skipping: [an-a01n02]
[2026-09-03T06:08:39.605Z] 
[2026-09-03T06:08:39.605Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-03T06:08:39.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-03T06:08:39.948Z] 
[2026-09-03T06:08:39.948Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T06:08:39.948Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-03T06:08:39.948Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-03T06:08:39.948Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-03T06:08:39.948Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-03T06:08:39.948Z]   echo "pcs:"
[2026-09-03T06:08:39.948Z]   echo "$pcsstatus"
[2026-09-03T06:08:39.948Z]   exit 1
[2026-09-03T06:08:39.948Z] fi
[2026-09-03T06:08:39.948Z] exit 0
[2026-09-03T06:08:39.948Z] ] ***
[2026-09-03T06:08:41.662Z] ok: [an-a01n01]
[2026-09-03T06:08:41.662Z] ok: [an-a01n02]
[2026-09-03T06:08:41.662Z] 
[2026-09-03T06:08:41.662Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-03T06:08:41.662Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-03T06:08:41.662Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-03T06:08:41.662Z] ret=0
[2026-09-03T06:08:41.662Z] if [ "$servers"  != "1" ]; then
[2026-09-03T06:08:41.662Z]   echo "Not all servers are running on the same node!"
[2026-09-03T06:08:41.662Z]   echo "$pcsstatus"
[2026-09-03T06:08:41.662Z]   ret=1
[2026-09-03T06:08:41.662Z] fi
[2026-09-03T06:08:41.662Z] echo "$hypervisor"
[2026-09-03T06:08:41.662Z] exit $ret
[2026-09-03T06:08:41.662Z] ] ***
[2026-09-03T06:08:43.983Z] ok: [an-a01n01]
[2026-09-03T06:08:43.983Z] ok: [an-a01n02]
[2026-09-03T06:08:43.983Z] 
[2026-09-03T06:08:43.983Z] PLAY [enable upgrade repo] *****************************************************
[2026-09-03T06:08:43.983Z] 
[2026-09-03T06:08:43.983Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:08:46.304Z] ok: [an-a01n02]
[2026-09-03T06:08:46.304Z] ok: [an-a01n01]
[2026-09-03T06:08:46.304Z] ok: [an-a01dr01]
[2026-09-03T06:08:46.304Z] ok: [an-striker02]
[2026-09-03T06:08:46.304Z] ok: [an-striker01]
[2026-09-03T06:08:46.304Z] 
[2026-09-03T06:08:46.304Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}')
[2026-09-03T06:08:46.304Z] dnf config-manager --set-enabled $updaterepo
[2026-09-03T06:08:46.304Z] ] ***
[2026-09-03T06:08:59.604Z] changed: [an-a01dr01]
[2026-09-03T06:08:59.604Z] changed: [an-a01n01]
[2026-09-03T06:08:59.604Z] changed: [an-a01n02]
[2026-09-03T06:08:59.604Z] changed: [an-striker02]
[2026-09-03T06:08:59.604Z] changed: [an-striker01]
[2026-09-03T06:08:59.604Z] 
[2026-09-03T06:08:59.604Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] ***
[2026-09-03T06:08:59.604Z] ok: [an-striker01]
[2026-09-03T06:08:59.604Z] ok: [an-striker02]
[2026-09-03T06:08:59.604Z] ok: [an-a01n01]
[2026-09-03T06:08:59.604Z] ok: [an-a01n02]
[2026-09-03T06:08:59.604Z] ok: [an-a01dr01]
[2026-09-03T06:08:59.604Z] 
[2026-09-03T06:08:59.604Z] PLAY [start upgrade process] ***************************************************
[2026-09-03T06:08:59.604Z] 
[2026-09-03T06:08:59.604Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:09:00.338Z] ok: [an-striker01]
[2026-09-03T06:09:00.338Z] 
[2026-09-03T06:09:00.338Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes
[2026-09-03T06:09:00.338Z] ] ***
[2026-09-03T06:41:20.579Z] changed: [an-striker01]
[2026-09-03T06:41:20.579Z] 
[2026-09-03T06:41:20.579Z] TASK [reboot striker01 reboot_timeout=1800] ************************************
[2026-09-03T06:41:54.247Z] changed: [an-striker01]
[2026-09-03T06:41:54.247Z] 
[2026-09-03T06:41:54.247Z] PLAY [reconnect all nodes] *****************************************************
[2026-09-03T06:41:54.247Z] 
[2026-09-03T06:41:54.247Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:41:54.247Z] ok: [an-a01n02]
[2026-09-03T06:41:54.247Z] ok: [an-a01dr01]
[2026-09-03T06:41:54.247Z] ok: [an-a01n01]
[2026-09-03T06:41:54.247Z] ok: [an-striker01]
[2026-09-03T06:41:54.247Z] ok: [an-striker02]
[2026-09-03T06:41:54.247Z] 
[2026-09-03T06:41:54.247Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] ***
[2026-09-03T06:41:54.247Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01
[2026-09-03T06:41:54.247Z] 
[2026-09-03T06:41:54.247Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-03T06:45:18.461Z] ok: [an-a01n02 -> localhost]
[2026-09-03T06:45:18.461Z] ok: [an-striker02 -> localhost]
[2026-09-03T06:45:18.461Z] ok: [an-striker01 -> localhost]
[2026-09-03T06:45:18.461Z] ok: [an-a01n01 -> localhost]
[2026-09-03T06:45:18.461Z] ok: [an-a01dr01 -> localhost]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then
[2026-09-03T06:45:18.461Z]   echo "Node current uptime higher than expected! tool failed to reboot nodes?"
[2026-09-03T06:45:18.461Z]   exit 1
[2026-09-03T06:45:18.461Z] fi
[2026-09-03T06:45:18.461Z] ] ***
[2026-09-03T06:45:18.461Z] changed: [an-a01n01]
[2026-09-03T06:45:18.461Z] changed: [an-a01dr01]
[2026-09-03T06:45:18.461Z] changed: [an-a01n02]
[2026-09-03T06:45:18.461Z] changed: [an-striker01]
[2026-09-03T06:45:18.461Z] changed: [an-striker02]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] PLAY [check all nodes status after upgrade] ************************************
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:45:18.461Z] ok: [an-a01n01]
[2026-09-03T06:45:18.461Z] ok: [an-a01n02]
[2026-09-03T06:45:18.461Z] ok: [an-a01dr01]
[2026-09-03T06:45:18.461Z] ok: [an-striker01]
[2026-09-03T06:45:18.461Z] ok: [an-striker02]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then
[2026-09-03T06:45:18.461Z]   echo "perl-Log-Journald update not found!"
[2026-09-03T06:45:18.461Z]   exit 1
[2026-09-03T06:45:18.461Z] fi
[2026-09-03T06:45:18.461Z] ] ***
[2026-09-03T06:45:18.461Z] changed: [an-a01n01]
[2026-09-03T06:45:18.461Z] changed: [an-a01dr01]
[2026-09-03T06:45:18.461Z] changed: [an-a01n02]
[2026-09-03T06:45:18.461Z] changed: [an-striker01]
[2026-09-03T06:45:18.461Z] changed: [an-striker02]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] PLAY [check nodes status after upgrade] ****************************************
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:45:18.461Z] ok: [an-a01n01]
[2026-09-03T06:45:18.461Z] ok: [an-a01n02]
[2026-09-03T06:45:18.461Z] ok: [an-a01dr01]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then
[2026-09-03T06:45:18.461Z]   echo "akmod-drbd update not found!"
[2026-09-03T06:45:18.461Z]   exit 1
[2026-09-03T06:45:18.461Z] fi
[2026-09-03T06:45:18.461Z] ] ***
[2026-09-03T06:45:18.461Z] changed: [an-a01n01]
[2026-09-03T06:45:18.461Z] changed: [an-a01dr01]
[2026-09-03T06:45:18.461Z] changed: [an-a01n02]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] PLAY [check hosted node status after upgrade] **********************************
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:45:18.461Z] ok: [an-a01n02]
[2026-09-03T06:45:18.461Z] ok: [an-a01n01]
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T06:45:18.461Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T06:45:18.461Z] 
[2026-09-03T06:45:18.461Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T06:45:18.461Z] ] ***
[2026-09-03T06:45:20.781Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T06:45:20.781Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T06:45:20.781Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T06:45:20.781Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T06:45:20.781Z] 
[2026-09-03T06:45:20.781Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] ***
[2026-09-03T06:45:20.781Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02
[2026-09-03T06:45:20.781Z] 
[2026-09-03T06:45:20.781Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-03T06:45:20.781Z] ] ***
[2026-09-03T06:45:22.492Z] ok: [an-a01n02]
[2026-09-03T06:45:22.492Z] ok: [an-a01n01]
[2026-09-03T06:45:22.492Z] 
[2026-09-03T06:45:22.492Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] ***
[2026-09-03T06:45:22.492Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02
[2026-09-03T06:45:22.492Z] 
[2026-09-03T06:45:22.492Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-03T06:45:22.492Z] ] ***
[2026-09-03T06:45:23.226Z] ok: [an-a01n01]
[2026-09-03T06:45:23.226Z] ok: [an-a01n02]
[2026-09-03T06:45:23.226Z] 
[2026-09-03T06:45:23.226Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] ***
[2026-09-03T06:45:23.226Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02
[2026-09-03T06:45:23.226Z] 
[2026-09-03T06:45:23.226Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-03T06:45:23.226Z] ] ***
[2026-09-03T06:45:23.959Z] ok: [an-a01n01]
[2026-09-03T06:45:23.959Z] ok: [an-a01n02]
[2026-09-03T06:45:23.959Z] 
[2026-09-03T06:45:23.959Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] ***
[2026-09-03T06:45:23.959Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02
[2026-09-03T06:45:23.959Z] 
[2026-09-03T06:45:23.959Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-03T06:45:23.959Z] ] ***
[2026-09-03T06:45:25.145Z] ok: [an-a01n01]
[2026-09-03T06:45:25.145Z] ok: [an-a01n02]
[2026-09-03T06:45:25.145Z] 
[2026-09-03T06:45:25.145Z] PLAY [test anvil package self update] ******************************************
[2026-09-03T06:45:25.145Z] 
[2026-09-03T06:45:25.145Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:45:28.181Z] ok: [an-a01n02]
[2026-09-03T06:45:28.181Z] ok: [an-a01dr01]
[2026-09-03T06:45:28.181Z] ok: [an-a01n01]
[2026-09-03T06:45:28.181Z] ok: [an-striker01]
[2026-09-03T06:45:28.181Z] ok: [an-striker02]
[2026-09-03T06:45:28.181Z] 
[2026-09-03T06:45:28.181Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] ***
[2026-09-03T06:45:33.041Z] changed: [an-a01n02] => (item=anvil-daemon.service)
[2026-09-03T06:45:33.041Z] changed: [an-a01dr01] => (item=anvil-daemon.service)
[2026-09-03T06:45:33.041Z] changed: [an-a01n01] => (item=anvil-daemon.service)
[2026-09-03T06:45:33.041Z] changed: [an-striker01] => (item=anvil-daemon.service)
[2026-09-03T06:45:33.041Z] changed: [an-a01n02] => (item=scancore.service)
[2026-09-03T06:45:33.041Z] changed: [an-a01n01] => (item=scancore.service)
[2026-09-03T06:45:33.041Z] changed: [an-a01dr01] => (item=scancore.service)
[2026-09-03T06:45:33.041Z] changed: [an-striker01] => (item=scancore.service)
[2026-09-03T06:45:33.041Z] changed: [an-striker02] => (item=anvil-daemon.service)
[2026-09-03T06:45:33.041Z] changed: [an-striker02] => (item=scancore.service)
[2026-09-03T06:45:33.041Z] 
[2026-09-03T06:45:33.041Z] PLAY [test striker package self update] ****************************************
[2026-09-03T06:45:33.041Z] 
[2026-09-03T06:45:33.041Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:45:34.228Z] ok: [an-striker01]
[2026-09-03T06:45:34.228Z] ok: [an-striker02]
[2026-09-03T06:45:34.228Z] 
[2026-09-03T06:45:34.228Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker
[2026-09-03T06:45:34.228Z] ] ***
[2026-09-03T06:46:02.188Z] changed: [an-striker01]
[2026-09-03T06:46:02.188Z] changed: [an-striker02]
[2026-09-03T06:46:02.188Z] 
[2026-09-03T06:46:02.188Z] PLAY [test node package self update] *******************************************
[2026-09-03T06:46:02.188Z] 
[2026-09-03T06:46:02.188Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:46:03.373Z] ok: [an-a01n02]
[2026-09-03T06:46:03.373Z] ok: [an-a01n01]
[2026-09-03T06:46:03.373Z] 
[2026-09-03T06:46:03.373Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node
[2026-09-03T06:46:03.373Z] ] ***
[2026-09-03T06:46:31.428Z] changed: [an-a01n02]
[2026-09-03T06:46:31.428Z] changed: [an-a01n01]
[2026-09-03T06:46:31.428Z] 
[2026-09-03T06:46:31.428Z] PLAY [test dr package self update] *********************************************
[2026-09-03T06:46:31.428Z] 
[2026-09-03T06:46:31.428Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:46:31.428Z] ok: [an-a01dr01]
[2026-09-03T06:46:31.428Z] 
[2026-09-03T06:46:31.428Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr
[2026-09-03T06:46:31.428Z] ] ***
[2026-09-03T06:46:59.393Z] changed: [an-a01dr01]
[2026-09-03T06:46:59.393Z] 
[2026-09-03T06:46:59.393Z] PLAY [test anvil package self update] ******************************************
[2026-09-03T06:46:59.393Z] 
[2026-09-03T06:46:59.393Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:46:59.393Z] ok: [an-striker02]
[2026-09-03T06:46:59.393Z] ok: [an-a01n02]
[2026-09-03T06:46:59.393Z] ok: [an-striker01]
[2026-09-03T06:46:59.393Z] ok: [an-a01n01]
[2026-09-03T06:46:59.393Z] ok: [an-a01dr01]
[2026-09-03T06:46:59.393Z] 
[2026-09-03T06:46:59.393Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] ***
[2026-09-03T06:47:01.712Z] changed: [an-a01n02] => (item=anvil-daemon.service)
[2026-09-03T06:47:01.712Z] changed: [an-a01n01] => (item=anvil-daemon.service)
[2026-09-03T06:47:01.712Z] changed: [an-a01dr01] => (item=anvil-daemon.service)
[2026-09-03T06:47:01.712Z] changed: [an-striker02] => (item=anvil-daemon.service)
[2026-09-03T06:47:01.712Z] changed: [an-striker01] => (item=anvil-daemon.service)
[2026-09-03T06:47:01.712Z] changed: [an-a01n01] => (item=scancore.service)
[2026-09-03T06:47:01.712Z] changed: [an-a01n02] => (item=scancore.service)
[2026-09-03T06:47:01.712Z] changed: [an-a01dr01] => (item=scancore.service)
[2026-09-03T06:47:01.712Z] changed: [an-striker02] => (item=scancore.service)
[2026-09-03T06:47:01.712Z] changed: [an-striker01] => (item=scancore.service)
[2026-09-03T06:47:01.712Z] 
[2026-09-03T06:47:01.712Z] PLAY [delete server tests] *****************************************************
[2026-09-03T06:47:01.712Z] 
[2026-09-03T06:47:01.712Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:47:03.423Z] ok: [an-a01n02]
[2026-09-03T06:47:03.423Z] ok: [an-a01n01]
[2026-09-03T06:47:03.423Z] 
[2026-09-03T06:47:03.423Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-03T06:47:03.765Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-03T06:47:03.765Z] 
[2026-09-03T06:47:03.765Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-03T06:47:03.765Z] ] ***
[2026-09-03T06:47:07.751Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-03T06:47:07.751Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-03T06:47:07.751Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-03T06:47:07.751Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-03T06:47:07.751Z] 
[2026-09-03T06:47:07.751Z] TASK [delete servers _raw_params=../../common/delete_server.yml] ***************
[2026-09-03T06:47:07.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-03T06:47:07.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-03T06:47:07.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-03T06:47:07.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-03T06:47:07.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-03T06:47:07.751Z] 
[2026-09-03T06:47:07.751Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:47:07.751Z] ] ***
[2026-09-03T06:47:09.461Z] skipping: [an-a01n01]
[2026-09-03T06:47:09.461Z] ok: [an-a01n02]
[2026-09-03T06:47:09.461Z] 
[2026-09-03T06:47:09.461Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T06:47:09.461Z]   --force \
[2026-09-03T06:47:09.461Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T06:47:09.461Z] ] ***
[2026-09-03T06:47:16.862Z] skipping: [an-a01n01]
[2026-09-03T06:47:16.863Z] changed: [an-a01n02]
[2026-09-03T06:47:16.863Z] 
[2026-09-03T06:47:16.863Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:47:16.863Z] ] ***
[2026-09-03T06:48:05.306Z] skipping: [an-a01n01]
[2026-09-03T06:48:05.306Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left).
[2026-09-03T06:48:05.306Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left).
[2026-09-03T06:48:05.306Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left).
[2026-09-03T06:48:05.306Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left).
[2026-09-03T06:48:05.306Z] ok: [an-a01n02]
[2026-09-03T06:48:05.306Z] 
[2026-09-03T06:48:05.306Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:48:05.306Z] ] ***
[2026-09-03T06:48:16.289Z] skipping: [an-a01n01]
[2026-09-03T06:48:16.289Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-03T06:48:16.289Z] ok: [an-a01n02]
[2026-09-03T06:48:16.289Z] 
[2026-09-03T06:48:16.289Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:48:16.289Z] ] ***
[2026-09-03T06:48:16.632Z] skipping: [an-a01n01]
[2026-09-03T06:48:16.632Z] ok: [an-a01n02]
[2026-09-03T06:48:16.632Z] 
[2026-09-03T06:48:16.632Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:16.632Z] ] ***
[2026-09-03T06:48:18.341Z] skipping: [an-a01n01]
[2026-09-03T06:48:18.341Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.913921", "end": "2026-09-03 02:48:18.034042", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 02:48:17.120121", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T06:48:18.341Z] ...ignoring
[2026-09-03T06:48:18.341Z] 
[2026-09-03T06:48:18.341Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T06:48:18.341Z]   --force \
[2026-09-03T06:48:18.341Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T06:48:18.341Z] ] ***
[2026-09-03T06:48:18.341Z] skipping: [an-a01n01]
[2026-09-03T06:48:18.341Z] skipping: [an-a01n02]
[2026-09-03T06:48:18.341Z] 
[2026-09-03T06:48:18.341Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:18.341Z] ] ***
[2026-09-03T06:48:18.341Z] skipping: [an-a01n01]
[2026-09-03T06:48:18.341Z] skipping: [an-a01n02]
[2026-09-03T06:48:18.341Z] 
[2026-09-03T06:48:18.341Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:48:18.341Z] ] ***
[2026-09-03T06:48:18.683Z] skipping: [an-a01n01]
[2026-09-03T06:48:18.683Z] skipping: [an-a01n02]
[2026-09-03T06:48:18.683Z] 
[2026-09-03T06:48:18.684Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:48:18.684Z] ] ***
[2026-09-03T06:48:18.684Z] skipping: [an-a01n01]
[2026-09-03T06:48:18.684Z] skipping: [an-a01n02]
[2026-09-03T06:48:18.684Z] 
[2026-09-03T06:48:18.684Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:18.684Z] ] ***
[2026-09-03T06:48:20.393Z] skipping: [an-a01n01]
[2026-09-03T06:48:20.393Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.979427", "end": "2026-09-03 02:48:19.970136", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 02:48:18.990709", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T06:48:20.393Z] ...ignoring
[2026-09-03T06:48:20.393Z] 
[2026-09-03T06:48:20.393Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T06:48:20.393Z]   --force \
[2026-09-03T06:48:20.393Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T06:48:20.393Z] ] ***
[2026-09-03T06:48:20.393Z] skipping: [an-a01n01]
[2026-09-03T06:48:20.393Z] skipping: [an-a01n02]
[2026-09-03T06:48:20.393Z] 
[2026-09-03T06:48:20.393Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:20.393Z] ] ***
[2026-09-03T06:48:20.393Z] skipping: [an-a01n01]
[2026-09-03T06:48:20.393Z] skipping: [an-a01n02]
[2026-09-03T06:48:20.393Z] 
[2026-09-03T06:48:20.393Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:48:20.393Z] ] ***
[2026-09-03T06:48:20.393Z] skipping: [an-a01n01]
[2026-09-03T06:48:20.393Z] skipping: [an-a01n02]
[2026-09-03T06:48:20.393Z] 
[2026-09-03T06:48:20.393Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:48:20.393Z] ] ***
[2026-09-03T06:48:20.393Z] skipping: [an-a01n01]
[2026-09-03T06:48:20.393Z] skipping: [an-a01n02]
[2026-09-03T06:48:20.393Z] 
[2026-09-03T06:48:20.393Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:20.393Z] ] ***
[2026-09-03T06:48:22.101Z] skipping: [an-a01n01]
[2026-09-03T06:48:22.101Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.932436", "end": "2026-09-03 02:48:21.819072", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 02:48:20.886636", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T06:48:22.101Z] ...ignoring
[2026-09-03T06:48:22.101Z] 
[2026-09-03T06:48:22.101Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T06:48:22.101Z]   --force \
[2026-09-03T06:48:22.101Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T06:48:22.101Z] ] ***
[2026-09-03T06:48:22.101Z] skipping: [an-a01n01]
[2026-09-03T06:48:22.101Z] skipping: [an-a01n02]
[2026-09-03T06:48:22.101Z] 
[2026-09-03T06:48:22.101Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:22.101Z] ] ***
[2026-09-03T06:48:22.101Z] skipping: [an-a01n01]
[2026-09-03T06:48:22.101Z] skipping: [an-a01n02]
[2026-09-03T06:48:22.101Z] 
[2026-09-03T06:48:22.101Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:48:22.101Z] ] ***
[2026-09-03T06:48:22.443Z] skipping: [an-a01n01]
[2026-09-03T06:48:22.443Z] skipping: [an-a01n02]
[2026-09-03T06:48:22.443Z] 
[2026-09-03T06:48:22.443Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:48:22.443Z] ] ***
[2026-09-03T06:48:22.443Z] skipping: [an-a01n01]
[2026-09-03T06:48:22.443Z] skipping: [an-a01n02]
[2026-09-03T06:48:22.443Z] 
[2026-09-03T06:48:22.443Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:22.443Z] ] ***
[2026-09-03T06:48:24.151Z] skipping: [an-a01n01]
[2026-09-03T06:48:24.151Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.955105", "end": "2026-09-03 02:48:23.675748", "msg": "non-zero return code", "rc": 1, "start": "2026-09-03 02:48:22.720643", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []}
[2026-09-03T06:48:24.151Z] ...ignoring
[2026-09-03T06:48:24.151Z] 
[2026-09-03T06:48:24.151Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-03T06:48:24.151Z]   --force \
[2026-09-03T06:48:24.151Z]   --server an-test-deploy{{ servernum }}
[2026-09-03T06:48:24.151Z] ] ***
[2026-09-03T06:48:24.151Z] skipping: [an-a01n01]
[2026-09-03T06:48:24.151Z] skipping: [an-a01n02]
[2026-09-03T06:48:24.151Z] 
[2026-09-03T06:48:24.151Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-03T06:48:24.151Z] ] ***
[2026-09-03T06:48:24.151Z] skipping: [an-a01n01]
[2026-09-03T06:48:24.151Z] skipping: [an-a01n02]
[2026-09-03T06:48:24.151Z] 
[2026-09-03T06:48:24.151Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-03T06:48:24.151Z] ] ***
[2026-09-03T06:48:24.151Z] skipping: [an-a01n01]
[2026-09-03T06:48:24.151Z] skipping: [an-a01n02]
[2026-09-03T06:48:24.151Z] 
[2026-09-03T06:48:24.151Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-03T06:48:24.151Z] ] ***
[2026-09-03T06:48:24.493Z] skipping: [an-a01n01]
[2026-09-03T06:48:24.493Z] skipping: [an-a01n02]
[2026-09-03T06:48:24.493Z] 
[2026-09-03T06:48:24.493Z] PLAY [Check anvil available resources from striker01] **************************
[2026-09-03T06:48:24.493Z] 
[2026-09-03T06:48:24.493Z] TASK [Gathering Facts ] ********************************************************
[2026-09-03T06:48:26.293Z] ok: [an-striker01]
[2026-09-03T06:48:26.293Z] 
[2026-09-03T06:48:26.293Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status"
[2026-09-03T06:48:26.293Z] sleep 300
[2026-09-03T06:48:26.293Z] anvil-report-usage > /root/anvil_available_resources.stop
[2026-09-03T06:48:26.293Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop
[2026-09-03T06:48:26.293Z] ] ***
[2026-09-03T06:53:36.149Z] ok: [an-striker01]
[2026-09-03T06:53:36.149Z] 
[2026-09-03T06:53:36.149Z] PLAY RECAP *********************************************************************
[2026-09-03T06:53:36.149Z] an-a01dr01                 : ok=90   changed=38   unreachable=0    failed=0    skipped=5    rescued=0    ignored=0   
[2026-09-03T06:53:36.149Z] an-a01n01                  : ok=314  changed=28   unreachable=0    failed=0    skipped=109  rescued=0    ignored=5   
[2026-09-03T06:53:36.149Z] an-a01n02                  : ok=345  changed=27   unreachable=0    failed=0    skipped=90   rescued=0    ignored=4   
[2026-09-03T06:53:36.149Z] an-striker01               : ok=57   changed=19   unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
[2026-09-03T06:53:36.149Z] an-striker02               : ok=42   changed=12   unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
[2026-09-03T06:53:36.149Z] 
[2026-09-03T06:53:36.149Z] tar: Removing leading `/' from member names
[2026-09-03T06:53:36.149Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-03T06:53:36.149Z] /root/ci-tools
[2026-09-03T06:53:36.149Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/1118.tar.gz
[2026-09-03T06:53:36.149Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat.
[2026-09-03T06:53:36.149Z] This might take a few seconds.
[2026-09-03T06:53:36.149Z] 
[2026-09-03T06:53:36.149Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T06:53:36.490Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T06:53:37.676Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T06:53:37.676Z] 
[2026-09-03T06:53:37.676Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T06:53:38.861Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T06:53:38.861Z] 
[2026-09-03T06:53:38.861Z] 1. Register with subscription-manager
[2026-09-03T06:53:38.861Z] # subscription-manager register
[2026-09-03T06:53:38.861Z] 
[2026-09-03T06:53:38.861Z] 2. Register with insights-client
[2026-09-03T06:53:38.861Z] # insights-client --register
[2026-09-03T06:53:40.048Z] This system is currently not registered.
[2026-09-03T06:53:40.391Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat.
[2026-09-03T06:53:40.391Z] This might take a few seconds.
[2026-09-03T06:53:40.391Z] 
[2026-09-03T06:53:40.733Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T06:53:43.055Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T06:53:45.375Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T06:53:45.375Z] 
[2026-09-03T06:53:45.375Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T06:53:47.085Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T06:53:47.085Z] 
[2026-09-03T06:53:47.085Z] 1. Register with subscription-manager
[2026-09-03T06:53:47.085Z] # subscription-manager register
[2026-09-03T06:53:47.085Z] 
[2026-09-03T06:53:47.085Z] 2. Register with insights-client
[2026-09-03T06:53:47.085Z] # insights-client --register
[2026-09-03T06:53:48.794Z] This system is currently not registered.
[2026-09-03T06:53:49.227Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat.
[2026-09-03T06:53:49.227Z] This might take a few seconds.
[2026-09-03T06:53:49.227Z] 
[2026-09-03T06:53:49.227Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T06:53:51.547Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T06:53:52.280Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T06:53:52.280Z] 
[2026-09-03T06:53:52.280Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T06:53:53.990Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T06:53:53.990Z] 
[2026-09-03T06:53:53.990Z] 1. Register with subscription-manager
[2026-09-03T06:53:53.990Z] # subscription-manager register
[2026-09-03T06:53:53.990Z] 
[2026-09-03T06:53:53.990Z] 2. Register with insights-client
[2026-09-03T06:53:53.990Z] # insights-client --register
[2026-09-03T06:53:54.724Z] This system is currently not registered.
[2026-09-03T06:53:55.066Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat.
[2026-09-03T06:53:55.066Z] This might take a few seconds.
[2026-09-03T06:53:55.066Z] 
[2026-09-03T06:53:55.066Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T06:53:57.386Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T06:53:58.120Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T06:53:58.120Z] 
[2026-09-03T06:53:58.120Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T06:53:59.306Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T06:53:59.306Z] 
[2026-09-03T06:53:59.306Z] 1. Register with subscription-manager
[2026-09-03T06:53:59.306Z] # subscription-manager register
[2026-09-03T06:53:59.306Z] 
[2026-09-03T06:53:59.306Z] 2. Register with insights-client
[2026-09-03T06:53:59.306Z] # insights-client --register
[2026-09-03T06:54:00.491Z] This system is currently not registered.
[2026-09-03T06:54:00.833Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat.
[2026-09-03T06:54:00.833Z] This might take a few seconds.
[2026-09-03T06:54:00.833Z] 
[2026-09-03T06:54:00.833Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-03T06:54:03.151Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-03T06:54:03.975Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-03T06:54:03.975Z] 
[2026-09-03T06:54:03.975Z] Manage your connected systems: https://red.ht/connector
[2026-09-03T06:54:05.161Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-03T06:54:05.161Z] 
[2026-09-03T06:54:05.161Z] 1. Register with subscription-manager
[2026-09-03T06:54:05.161Z] # subscription-manager register
[2026-09-03T06:54:05.161Z] 
[2026-09-03T06:54:05.161Z] 2. Register with insights-client
[2026-09-03T06:54:05.161Z] # insights-client --register
[2026-09-03T06:54:06.346Z] This system is currently not registered.
[2026-09-03T06:54:06.416Z] [Pipeline] }
[2026-09-03T06:54:06.420Z] [Pipeline] // timeout
[2026-09-03T06:54:06.425Z] [Pipeline] }
[2026-09-03T06:54:06.427Z] [Pipeline] // script
[2026-09-03T06:54:06.431Z] [Pipeline] }
[2026-09-03T06:54:06.433Z] [Pipeline] // stage
[2026-09-03T06:54:06.439Z] [Pipeline] stage
[2026-09-03T06:54:06.439Z] [Pipeline] { (Declarative: Post Actions)
[2026-09-03T06:54:06.447Z] [Pipeline] cleanWs
[2026-09-03T06:54:06.496Z] [WS-CLEANUP] Deleting project workspace...
[2026-09-03T06:54:06.496Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[2026-09-03T06:54:06.520Z] [WS-CLEANUP] done
[2026-09-03T06:54:06.524Z] [Pipeline] script
[2026-09-03T06:54:06.528Z] [Pipeline] {
[2026-09-03T06:54:06.553Z] [Pipeline] stage
[2026-09-03T06:54:06.553Z] [Pipeline] { (Copying logs)
[2026-09-03T06:54:06.595Z] [Pipeline] node
[2026-09-03T06:54:06.703Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing
[2026-09-03T06:54:06.704Z] [Pipeline] {
[2026-09-03T06:54:06.755Z] [Pipeline] node
[2026-09-03T06:54:06.870Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2
[2026-09-03T06:54:06.871Z] [Pipeline] {
[2026-09-03T06:54:06.875Z] [Pipeline] sh
[2026-09-03T06:54:07.131Z] + mkdir -p /var/lib/jenkins/locks
[2026-09-03T06:54:07.145Z] [Pipeline] echo
[2026-09-03T06:54:07.145Z] RWLock: FD for lock log_archive in stage projectFinishUp is 430
[2026-09-03T06:54:07.149Z] [Pipeline] sleep
[2026-09-03T06:54:07.149Z] No need to sleep any longer
[2026-09-03T06:54:07.152Z] [Pipeline] echo
[2026-09-03T06:54:07.152Z] RWLock: log_archive in stage projectFinishUp locked for WRITE
[2026-09-03T06:54:07.156Z] [Pipeline] }
[2026-09-03T06:54:07.159Z] [Pipeline] // node
[2026-09-03T06:54:07.163Z] [Pipeline] echo
[2026-09-03T06:54:07.163Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1118/
[2026-09-03T06:54:07.164Z] [Pipeline] sh
[2026-09-03T06:54:07.419Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1118/
[2026-09-03T06:54:07.421Z] [Pipeline] sh