[2026-09-29T08:50:42.825Z] Started by upstream project "anvil/anvil-build-cron" build number 180
[2026-09-29T08:50:42.825Z] originally caused by:
[2026-09-29T08:50:42.825Z]  Started by timer
[2026-09-29T08:50:43.311Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools
[2026-09-29T08:50:43.321Z] Loading library ProjectLib@main:pipelines/projects/anvil/
[2026-09-29T08:50:43.703Z] Examining kronosnet/ci-tools
[2026-09-29T08:50:43.703Z] Attempting to resolve main as a branch
[2026-09-29T08:50:43.901Z] Resolved main as branch main at revision b175ea1eaa3b116a3499f67254edeed28a1abb28
[2026-09-29T08:50:43.902Z] Selected Git installation does not exist. Using Default
[2026-09-29T08:50:43.902Z] The recommended git tool is: NONE
[2026-09-29T08:50:43.902Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9
[2026-09-29T08:50:43.905Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead.
[2026-09-29T08:50:43.905Z] Wiping out workspace first.
[2026-09-29T08:50:43.930Z] Cloning the remote Git repository
[2026-09-29T08:50:43.930Z] Using shallow clone with depth 1
[2026-09-29T08:50:43.930Z] Avoid fetching tags
[2026-09-29T08:50:43.930Z] Honoring refspec on initial clone
[2026-09-29T08:50:43.931Z] Cloning repository https://github.com/kronosnet/ci-tools.git
[2026-09-29T08:50:43.931Z]  > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10
[2026-09-29T08:50:43.940Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git
[2026-09-29T08:50:43.940Z]  > git --version # timeout=10
[2026-09-29T08:50:43.947Z]  > git --version # 'git version 2.52.0'
[2026-09-29T08:50:43.947Z] using GIT_ASKPASS to set credentials getCredUUID - github
[2026-09-29T08:50:43.955Z]  > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-29T08:50:44.323Z]  > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10
[2026-09-29T08:50:44.330Z]  > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-29T08:50:44.339Z] Avoid second fetch
[2026-09-29T08:50:44.339Z] Checking out Revision b175ea1eaa3b116a3499f67254edeed28a1abb28 (main)
[2026-09-29T08:50:44.339Z]  > git config core.sparsecheckout # timeout=10
[2026-09-29T08:50:44.347Z]  > git checkout -f b175ea1eaa3b116a3499f67254edeed28a1abb28 # timeout=10
[2026-09-29T08:50:44.381Z] Commit message: "support ansible development/testing in CITBRANCH"
[2026-09-29T08:50:44.390Z] Loading library GlobalLib@main:./pipelines/libs/global/
[2026-09-29T08:50:44.618Z] Examining kronosnet/ci-tools
[2026-09-29T08:50:44.618Z] Attempting to resolve main as a branch
[2026-09-29T08:50:44.782Z] Resolved main as branch main at revision b175ea1eaa3b116a3499f67254edeed28a1abb28
[2026-09-29T08:50:44.782Z] Selected Git installation does not exist. Using Default
[2026-09-29T08:50:44.782Z] The recommended git tool is: NONE
[2026-09-29T08:50:44.783Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9
[2026-09-29T08:50:44.786Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead.
[2026-09-29T08:50:44.786Z] Wiping out workspace first.
[2026-09-29T08:50:44.816Z] Cloning the remote Git repository
[2026-09-29T08:50:44.816Z] Using shallow clone with depth 1
[2026-09-29T08:50:44.816Z] Avoid fetching tags
[2026-09-29T08:50:44.816Z] Honoring refspec on initial clone
[2026-09-29T08:50:44.816Z] Cloning repository https://github.com/kronosnet/ci-tools.git
[2026-09-29T08:50:44.816Z]  > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@4 # timeout=10
[2026-09-29T08:50:44.837Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git
[2026-09-29T08:50:44.837Z]  > git --version # timeout=10
[2026-09-29T08:50:44.846Z]  > git --version # 'git version 2.52.0'
[2026-09-29T08:50:44.846Z] using GIT_ASKPASS to set credentials getCredUUID - github
[2026-09-29T08:50:44.862Z]  > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-29T08:50:45.230Z]  > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10
[2026-09-29T08:50:45.238Z]  > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10
[2026-09-29T08:50:45.246Z] Avoid second fetch
[2026-09-29T08:50:45.246Z] Checking out Revision b175ea1eaa3b116a3499f67254edeed28a1abb28 (main)
[2026-09-29T08:50:45.246Z]  > git config core.sparsecheckout # timeout=10
[2026-09-29T08:50:45.253Z]  > git checkout -f b175ea1eaa3b116a3499f67254edeed28a1abb28 # timeout=10
[2026-09-29T08:50:45.291Z] Commit message: "support ansible development/testing in CITBRANCH"
[2026-09-29T08:50:45.390Z] [Pipeline] Start of Pipeline
[2026-09-29T08:50:45.485Z] [Pipeline] node
[2026-09-29T08:50:45.491Z] Running on anvil-ci-bm-phy07 in /srv/workspace/anvil/anvil-build-functional-testing
[2026-09-29T08:50:45.492Z] [Pipeline] {
[2026-09-29T08:50:45.502Z] [Pipeline] stage
[2026-09-29T08:50:45.502Z] [Pipeline] { (Prep build env)
[2026-09-29T08:50:45.517Z] [Pipeline] script
[2026-09-29T08:50:45.519Z] [Pipeline] {
[2026-09-29T08:50:45.523Z] [Pipeline] cleanWs
[2026-09-29T08:50:45.607Z] [WS-CLEANUP] Deleting project workspace...
[2026-09-29T08:50:45.607Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[2026-09-29T08:50:45.648Z] [WS-CLEANUP] done
[2026-09-29T08:50:45.672Z] [Pipeline] lock
[2026-09-29T08:50:45.672Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT]
[2026-09-29T08:50:45.673Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created.
[2026-09-29T08:50:45.673Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT]
[2026-09-29T08:50:45.673Z] [Pipeline] {
[2026-09-29T08:50:45.676Z] [Pipeline] }
[2026-09-29T08:50:45.676Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT]
[2026-09-29T08:50:45.679Z] [Pipeline] // lock
[2026-09-29T08:50:45.683Z] [Pipeline] }
[2026-09-29T08:50:45.685Z] [Pipeline] // script
[2026-09-29T08:50:45.687Z] [Pipeline] }
[2026-09-29T08:50:45.689Z] [Pipeline] // stage
[2026-09-29T08:50:45.692Z] [Pipeline] stage
[2026-09-29T08:50:45.693Z] [Pipeline] { (Run functional testing)
[2026-09-29T08:50:45.702Z] [Pipeline] script
[2026-09-29T08:50:45.703Z] [Pipeline] {
[2026-09-29T08:50:45.729Z] [Pipeline] echo
[2026-09-29T08:50:45.729Z] runWithTimeout 360
[2026-09-29T08:50:45.730Z] [Pipeline] timeout
[2026-09-29T08:50:45.730Z] Timeout set to expire in 6 hr 0 min
[2026-09-29T08:50:45.730Z] [Pipeline] {
[2026-09-29T08:50:45.735Z] [Pipeline] sh
[2026-09-29T08:50:46.353Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 ci ci all
[2026-09-29T08:50:46.353Z] CITBRANCH:  main
[2026-09-29T08:50:46.353Z] CITHOME:    /root/ci-tools
[2026-09-29T08:50:46.353Z] CITCMD:     ci-setup-anvil-bm-vm
[2026-09-29T08:50:46.353Z] CITCMDARGS: rhel-9 ci ci all
[2026-09-29T08:50:46.353Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-29T08:50:46.353Z] Genering ssh keys and ssh config
[2026-09-29T08:50:47.222Z] Delay build by 180 seconds to allow Jenkins to cancel the job
[2026-09-29T08:53:58.272Z] Destroying VMs (if any)
[2026-09-29T08:53:58.272Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-29T08:53:58.272Z] Genering ssh keys and ssh config
[2026-09-29T08:53:58.272Z] Cache already updated
[2026-09-29T08:53:58.272Z] Creating nodes disks
[2026-09-29T08:53:58.272Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16
[2026-09-29T08:53:58.272Z] Generating an-test-deployX kickstart files/isos
[2026-09-29T08:53:58.272Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso'
[2026-09-29T08:53:58.272Z] Media current: stdio file, overwriteable
[2026-09-29T08:53:58.272Z] Media status : is blank
[2026-09-29T08:53:58.272Z] Media summary: 0 sessions, 0 data blocks, 0 data,  791g free
[2026-09-29T08:53:58.272Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1'
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] ISO image produced: 184 sectors
[2026-09-29T08:53:58.272Z] Written to medium : 184 sectors at LBA 0
[2026-09-29T08:53:58.272Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso'
[2026-09-29T08:53:58.272Z] Media current: stdio file, overwriteable
[2026-09-29T08:53:58.272Z] Media status : is blank
[2026-09-29T08:53:58.272Z] Media summary: 0 sessions, 0 data blocks, 0 data,  801g free
[2026-09-29T08:53:58.272Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2'
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] ISO image produced: 184 sectors
[2026-09-29T08:53:58.272Z] Written to medium : 184 sectors at LBA 0
[2026-09-29T08:53:58.272Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso'
[2026-09-29T08:53:58.272Z] Media current: stdio file, overwriteable
[2026-09-29T08:53:58.272Z] Media status : is blank
[2026-09-29T08:53:58.272Z] Media summary: 0 sessions, 0 data blocks, 0 data,  801g free
[2026-09-29T08:53:58.272Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3'
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] ISO image produced: 184 sectors
[2026-09-29T08:53:58.272Z] Written to medium : 184 sectors at LBA 0
[2026-09-29T08:53:58.272Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso'
[2026-09-29T08:53:58.272Z] Media current: stdio file, overwriteable
[2026-09-29T08:53:58.272Z] Media status : is blank
[2026-09-29T08:53:58.272Z] Media summary: 0 sessions, 0 data blocks, 0 data,  801g free
[2026-09-29T08:53:58.272Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4'
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] ISO image produced: 184 sectors
[2026-09-29T08:53:58.272Z] Written to medium : 184 sectors at LBA 0
[2026-09-29T08:53:58.272Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso'
[2026-09-29T08:53:58.272Z] Media current: stdio file, overwriteable
[2026-09-29T08:53:58.272Z] Media status : is blank
[2026-09-29T08:53:58.272Z] Media summary: 0 sessions, 0 data blocks, 0 data,  801g free
[2026-09-29T08:53:58.272Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5'
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] xorriso : UPDATE :       1 files added in 1 seconds
[2026-09-29T08:53:58.272Z] ISO image produced: 184 sectors
[2026-09-29T08:53:58.272Z] Written to medium : 184 sectors at LBA 0
[2026-09-29T08:53:58.272Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Generating cloud-init configs
[2026-09-29T08:53:58.272Z] Generating cloud-init metadata and seed images
[2026-09-29T08:53:58.272Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-29T08:53:58.272Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-29T08:53:58.272Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-29T08:53:58.272Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-29T08:53:58.272Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw
[2026-09-29T08:53:58.272Z] Creating VMs
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Starting install...
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Creating domain...                                          |         00:00     
[2026-09-29T08:53:58.272Z] Domain creation completed.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Starting install...
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Creating domain...                                          |         00:00     
[2026-09-29T08:53:58.272Z] Domain creation completed.
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Starting install...
[2026-09-29T08:53:58.272Z] 
[2026-09-29T08:53:58.272Z] Creating domain...                                          |         00:00     
[2026-09-29T08:53:58.272Z] Domain creation completed.
[2026-09-29T08:53:59.142Z] 
[2026-09-29T08:53:59.142Z] Starting install...
[2026-09-29T08:53:59.142Z] 
[2026-09-29T08:53:59.142Z] Creating domain...                                          |         00:00     
[2026-09-29T08:53:59.142Z] Domain creation completed.
[2026-09-29T08:54:00.531Z] 
[2026-09-29T08:54:00.531Z] Starting install...
[2026-09-29T08:54:00.531Z] 
[2026-09-29T08:54:00.531Z] Creating domain...                                          |         00:00     
[2026-09-29T08:54:00.531Z] Domain creation completed.
[2026-09-29T08:54:00.531Z] Waiting for nodes and cloud-init to complete full init
[2026-09-29T08:54:04.518Z] cloud-init status node an-striker01: 
[2026-09-29T08:54:07.531Z] cloud-init status node an-striker02: 
[2026-09-29T08:54:10.194Z] cloud-init status node an-a01n01: 
[2026-09-29T08:54:13.637Z] cloud-init status node an-a01n02: 
[2026-09-29T08:54:17.081Z] cloud-init status node an-a01dr01: 
[2026-09-29T08:54:17.081Z] Tue Sep 29 04:54:16 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:54:29.230Z] cloud-init status node an-striker01: running
[2026-09-29T08:54:29.230Z] cloud-init status node an-striker02: running
[2026-09-29T08:54:30.328Z] cloud-init status node an-a01n01: running
[2026-09-29T08:54:31.898Z] cloud-init status node an-a01n02: running
[2026-09-29T08:54:32.831Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:54:32.831Z] Tue Sep 29 04:54:32 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:54:44.225Z] cloud-init status node an-striker01: running
[2026-09-29T08:54:44.769Z] cloud-init status node an-striker02: running
[2026-09-29T08:54:45.398Z] cloud-init status node an-a01n01: running
[2026-09-29T08:54:51.444Z] cloud-init status node an-a01n02: running
[2026-09-29T08:54:52.049Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:54:52.049Z] Tue Sep 29 04:54:51 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:55:03.899Z] cloud-init status node an-striker01: running
[2026-09-29T08:55:03.899Z] cloud-init status node an-striker02: running
[2026-09-29T08:55:05.279Z] cloud-init status node an-a01n01: running
[2026-09-29T08:55:05.927Z] cloud-init status node an-a01n02: running
[2026-09-29T08:55:06.614Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:55:06.614Z] Tue Sep 29 04:55:06 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:55:17.832Z] cloud-init status node an-striker01: running
[2026-09-29T08:55:19.066Z] cloud-init status node an-striker02: running
[2026-09-29T08:55:19.488Z] cloud-init status node an-a01n01: running
[2026-09-29T08:55:20.421Z] cloud-init status node an-a01n02: running
[2026-09-29T08:55:21.355Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:55:21.355Z] Tue Sep 29 04:55:21 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:55:34.867Z] cloud-init status node an-striker01: running
[2026-09-29T08:55:34.867Z] cloud-init status node an-striker02: running
[2026-09-29T08:55:34.867Z] cloud-init status node an-a01n01: running
[2026-09-29T08:55:45.334Z] cloud-init status node an-a01n02: running
[2026-09-29T08:55:46.210Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:55:46.210Z] Tue Sep 29 04:55:46 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:55:58.074Z] cloud-init status node an-striker01: running
[2026-09-29T08:55:58.074Z] cloud-init status node an-striker02: running
[2026-09-29T08:55:59.466Z] cloud-init status node an-a01n01: running
[2026-09-29T08:56:00.337Z] cloud-init status node an-a01n02: running
[2026-09-29T08:56:01.220Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:56:01.220Z] Tue Sep 29 04:56:00 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:56:13.017Z] cloud-init status node an-striker01: running
[2026-09-29T08:56:13.017Z] cloud-init status node an-striker02: running
[2026-09-29T08:56:14.408Z] cloud-init status node an-a01n01: running
[2026-09-29T08:56:21.043Z] cloud-init status node an-a01n02: running
[2026-09-29T08:56:21.043Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:56:21.043Z] Tue Sep 29 04:56:20 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:56:33.153Z] cloud-init status node an-striker01: running
[2026-09-29T08:56:38.555Z] cloud-init status node an-striker02: running
[2026-09-29T08:56:38.965Z] cloud-init status node an-a01n01: running
[2026-09-29T08:56:40.355Z] cloud-init status node an-a01n02: running
[2026-09-29T08:56:41.225Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:56:41.225Z] Tue Sep 29 04:56:40 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:56:53.024Z] cloud-init status node an-striker01: running
[2026-09-29T08:56:53.434Z] cloud-init status node an-striker02: running
[2026-09-29T08:56:54.312Z] cloud-init status node an-a01n01: running
[2026-09-29T08:56:55.212Z] cloud-init status node an-a01n02: running
[2026-09-29T08:56:56.086Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:56:56.086Z] Tue Sep 29 04:56:55 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:57:07.934Z] cloud-init status node an-striker01: running
[2026-09-29T08:57:07.934Z] cloud-init status node an-striker02: running
[2026-09-29T08:57:08.806Z] cloud-init status node an-a01n01: running
[2026-09-29T08:57:09.678Z] cloud-init status node an-a01n02: running
[2026-09-29T08:57:10.548Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:57:10.548Z] Tue Sep 29 04:57:10 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:57:22.349Z] cloud-init status node an-striker01: running
[2026-09-29T08:57:22.349Z] cloud-init status node an-striker02: running
[2026-09-29T08:57:23.750Z] cloud-init status node an-a01n01: running
[2026-09-29T08:57:24.622Z] cloud-init status node an-a01n02: running
[2026-09-29T08:57:25.517Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:57:25.517Z] Tue Sep 29 04:57:25 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:57:37.306Z] cloud-init status node an-striker01: running
[2026-09-29T08:57:37.306Z] cloud-init status node an-striker02: running
[2026-09-29T08:57:38.177Z] cloud-init status node an-a01n01: running
[2026-09-29T08:57:39.048Z] cloud-init status node an-a01n02: running
[2026-09-29T08:57:39.918Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:57:39.918Z] Tue Sep 29 04:57:39 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:57:51.711Z] cloud-init status node an-striker01: running
[2026-09-29T08:57:51.711Z] cloud-init status node an-striker02: running
[2026-09-29T08:57:52.581Z] cloud-init status node an-a01n01: running
[2026-09-29T08:57:53.730Z] cloud-init status node an-a01n02: running
[2026-09-29T08:57:54.677Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:57:54.677Z] Tue Sep 29 04:57:54 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:58:06.469Z] cloud-init status node an-striker01: running
[2026-09-29T08:58:06.469Z] cloud-init status node an-striker02: running
[2026-09-29T08:58:07.338Z] cloud-init status node an-a01n01: running
[2026-09-29T08:58:08.208Z] cloud-init status node an-a01n02: running
[2026-09-29T08:58:09.599Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:58:09.599Z] Tue Sep 29 04:58:09 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:58:21.401Z] cloud-init status node an-striker01: running
[2026-09-29T08:58:21.401Z] cloud-init status node an-striker02: running
[2026-09-29T08:58:22.272Z] cloud-init status node an-a01n01: running
[2026-09-29T08:58:22.917Z] cloud-init status node an-a01n02: running
[2026-09-29T08:58:24.307Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:58:24.307Z] Tue Sep 29 04:58:23 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:58:36.103Z] cloud-init status node an-striker01: running
[2026-09-29T08:58:36.513Z] cloud-init status node an-striker02: running
[2026-09-29T08:58:37.383Z] cloud-init status node an-a01n01: running
[2026-09-29T08:58:38.774Z] cloud-init status node an-a01n02: running
[2026-09-29T08:58:39.645Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:58:39.645Z] Tue Sep 29 04:58:39 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:58:51.757Z] cloud-init status node an-striker01: running
[2026-09-29T08:58:51.757Z] cloud-init status node an-striker02: running
[2026-09-29T08:58:52.627Z] cloud-init status node an-a01n01: running
[2026-09-29T08:58:53.498Z] cloud-init status node an-a01n02: running
[2026-09-29T08:58:54.368Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:58:54.368Z] Tue Sep 29 04:58:54 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:59:06.166Z] cloud-init status node an-striker01: running
[2026-09-29T08:59:06.166Z] cloud-init status node an-striker02: running
[2026-09-29T08:59:07.036Z] cloud-init status node an-a01n01: running
[2026-09-29T08:59:08.426Z] cloud-init status node an-a01n02: running
[2026-09-29T08:59:09.297Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:59:09.297Z] Tue Sep 29 04:59:08 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:59:21.367Z] cloud-init status node an-striker01: running
[2026-09-29T08:59:21.367Z] cloud-init status node an-striker02: running
[2026-09-29T08:59:22.239Z] cloud-init status node an-a01n01: running
[2026-09-29T08:59:23.110Z] cloud-init status node an-a01n02: running
[2026-09-29T08:59:23.981Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:59:23.981Z] Tue Sep 29 04:59:23 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:59:35.774Z] cloud-init status node an-striker01: running
[2026-09-29T08:59:35.774Z] cloud-init status node an-striker02: running
[2026-09-29T08:59:37.165Z] cloud-init status node an-a01n01: running
[2026-09-29T08:59:38.036Z] cloud-init status node an-a01n02: running
[2026-09-29T08:59:38.906Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:59:38.906Z] Tue Sep 29 04:59:38 EDT 2026 actual: 0 expected: 5
[2026-09-29T08:59:50.942Z] cloud-init status node an-striker01: running
[2026-09-29T08:59:50.942Z] cloud-init status node an-striker02: running
[2026-09-29T08:59:51.814Z] cloud-init status node an-a01n01: running
[2026-09-29T08:59:52.684Z] cloud-init status node an-a01n02: running
[2026-09-29T08:59:53.554Z] cloud-init status node an-a01dr01: running
[2026-09-29T08:59:53.554Z] Tue Sep 29 04:59:53 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:00:05.359Z] cloud-init status node an-striker01: running
[2026-09-29T09:00:07.463Z] cloud-init status node an-striker02: running
[2026-09-29T09:00:08.333Z] cloud-init status node an-a01n01: running
[2026-09-29T09:00:09.203Z] cloud-init status node an-a01n02: running
[2026-09-29T09:00:10.076Z] cloud-init status node an-a01dr01: running
[2026-09-29T09:00:10.076Z] Tue Sep 29 05:00:09 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:00:21.261Z] cloud-init status node an-striker01: running
[2026-09-29T09:00:21.673Z] cloud-init status node an-striker02: running
[2026-09-29T09:00:22.545Z] cloud-init status node an-a01n01: running
[2026-09-29T09:00:23.934Z] cloud-init status node an-a01n02: running
[2026-09-29T09:00:24.804Z] cloud-init status node an-a01dr01: running
[2026-09-29T09:00:24.804Z] Tue Sep 29 05:00:24 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:00:36.604Z] cloud-init status node an-striker01: running
[2026-09-29T09:00:36.604Z] cloud-init status node an-striker02: running
[2026-09-29T09:00:37.994Z] cloud-init status node an-a01n01: running
[2026-09-29T09:00:38.864Z] cloud-init status node an-a01n02: running
[2026-09-29T09:00:39.736Z] cloud-init status node an-a01dr01: running
[2026-09-29T09:00:39.736Z] Tue Sep 29 05:00:39 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:00:52.311Z] cloud-init status node an-striker01: running
[2026-09-29T09:00:52.311Z] cloud-init status node an-striker02: running
[2026-09-29T09:00:52.311Z] cloud-init status node an-a01n01: running
[2026-09-29T09:00:53.182Z] cloud-init status node an-a01n02: running
[2026-09-29T09:00:54.057Z] cloud-init status node an-a01dr01: running
[2026-09-29T09:00:54.057Z] Tue Sep 29 05:00:53 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:01:05.855Z] cloud-init status node an-striker01: running
[2026-09-29T09:01:06.728Z] cloud-init status node an-striker02: running
[2026-09-29T09:01:08.120Z] cloud-init status node an-a01n01: running
[2026-09-29T09:01:08.991Z] cloud-init status node an-a01n02: running
[2026-09-29T09:01:10.366Z] cloud-init status node an-a01dr01: running
[2026-09-29T09:01:10.366Z] Tue Sep 29 05:01:09 EDT 2026 actual: 0 expected: 5
[2026-09-29T09:01:22.170Z] cloud-init status node an-striker01: running
[2026-09-29T09:01:22.581Z] cloud-init status node an-striker02: running
[2026-09-29T09:01:23.972Z] cloud-init status node an-a01n01: running
[2026-09-29T09:01:24.844Z] cloud-init status node an-a01n02: done
[2026-09-29T09:01:25.716Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:01:25.717Z] Tue Sep 29 05:01:25 EDT 2026 actual: 2 expected: 5
[2026-09-29T09:01:37.519Z] cloud-init status node an-striker01: running
[2026-09-29T09:01:39.341Z] cloud-init status node an-striker02: running
[2026-09-29T09:01:39.751Z] cloud-init status node an-a01n01: done
[2026-09-29T09:01:40.622Z] cloud-init status node an-a01n02: done
[2026-09-29T09:01:41.494Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:01:41.494Z] Tue Sep 29 05:01:41 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:01:53.290Z] cloud-init status node an-striker01: running
[2026-09-29T09:01:53.700Z] cloud-init status node an-striker02: running
[2026-09-29T09:01:54.571Z] cloud-init status node an-a01n01: done
[2026-09-29T09:01:55.441Z] cloud-init status node an-a01n02: done
[2026-09-29T09:01:56.831Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:01:56.831Z] Tue Sep 29 05:01:56 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:02:08.900Z] cloud-init status node an-striker01: running
[2026-09-29T09:02:08.900Z] cloud-init status node an-striker02: running
[2026-09-29T09:02:09.768Z] cloud-init status node an-a01n01: done
[2026-09-29T09:02:10.639Z] cloud-init status node an-a01n02: done
[2026-09-29T09:02:11.511Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:02:11.511Z] Tue Sep 29 05:02:11 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:02:23.308Z] cloud-init status node an-striker01: running
[2026-09-29T09:02:23.718Z] cloud-init status node an-striker02: running
[2026-09-29T09:02:24.590Z] cloud-init status node an-a01n01: done
[2026-09-29T09:02:25.461Z] cloud-init status node an-a01n02: done
[2026-09-29T09:02:26.332Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:02:26.332Z] Tue Sep 29 05:02:26 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:02:38.134Z] cloud-init status node an-striker01: running
[2026-09-29T09:02:38.544Z] cloud-init status node an-striker02: running
[2026-09-29T09:02:39.934Z] cloud-init status node an-a01n01: done
[2026-09-29T09:02:40.805Z] cloud-init status node an-a01n02: done
[2026-09-29T09:02:41.676Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:02:41.676Z] Tue Sep 29 05:02:41 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:02:53.475Z] cloud-init status node an-striker01: running
[2026-09-29T09:02:53.885Z] cloud-init status node an-striker02: running
[2026-09-29T09:02:54.755Z] cloud-init status node an-a01n01: done
[2026-09-29T09:02:56.147Z] cloud-init status node an-a01n02: done
[2026-09-29T09:02:57.018Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:02:57.018Z] Tue Sep 29 05:02:56 EDT 2026 actual: 3 expected: 5
[2026-09-29T09:03:09.130Z] cloud-init status node an-striker01: done
[2026-09-29T09:03:09.130Z] cloud-init status node an-striker02: done
[2026-09-29T09:03:10.001Z] cloud-init status node an-a01n01: done
[2026-09-29T09:03:10.873Z] cloud-init status node an-a01n02: done
[2026-09-29T09:03:11.744Z] cloud-init status node an-a01dr01: done
[2026-09-29T09:03:11.744Z] Tue Sep 29 05:03:11 EDT 2026 actual: 5 expected: 5
[2026-09-29T09:03:11.744Z] Building simengine model
[2026-09-29T09:03:25.926Z] Generating anvil striker auto config
[2026-09-29T09:03:28.590Z] Cloning into 'anvil-test-suite'...
[2026-09-29T09:03:29.698Z] generated ansible config:
[2026-09-29T09:03:29.698Z] =========================
[2026-09-29T09:03:29.698Z] [defaults]
[2026-09-29T09:03:29.698Z] inventory = ansible-inventory
[2026-09-29T09:03:29.698Z] forks = 500
[2026-09-29T09:03:29.698Z] any_errors_fatal = true
[2026-09-29T09:03:29.698Z] display_args_to_stdout = true
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:29.698Z] [ssh_connection]
[2026-09-29T09:03:29.698Z] retries = 20
[2026-09-29T09:03:29.698Z] =========================
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:29.698Z] generated ansible inventory:
[2026-09-29T09:03:29.698Z] ============================
[2026-09-29T09:03:29.698Z] [strikers]
[2026-09-29T09:03:29.698Z] an-striker01
[2026-09-29T09:03:29.698Z] an-striker02
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:29.698Z] [nodes]
[2026-09-29T09:03:29.698Z] an-a01n01
[2026-09-29T09:03:29.698Z] an-a01n02
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:29.698Z] [drnodes]
[2026-09-29T09:03:29.698Z] an-a01dr01
[2026-09-29T09:03:29.698Z] ============================
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:29.698Z] Checking connectivity to nodes
[2026-09-29T09:03:29.698Z] 
[2026-09-29T09:03:31.720Z] an-a01dr01 | SUCCESS => {
[2026-09-29T09:03:31.720Z]     "ansible_facts": {
[2026-09-29T09:03:31.720Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-29T09:03:31.720Z]     },
[2026-09-29T09:03:31.720Z]     "changed": false,
[2026-09-29T09:03:31.720Z]     "ping": "pong"
[2026-09-29T09:03:31.720Z] }
[2026-09-29T09:03:31.720Z] an-a01n02 | SUCCESS => {
[2026-09-29T09:03:31.720Z]     "ansible_facts": {
[2026-09-29T09:03:31.720Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-29T09:03:31.720Z]     },
[2026-09-29T09:03:31.720Z]     "changed": false,
[2026-09-29T09:03:31.720Z]     "ping": "pong"
[2026-09-29T09:03:31.720Z] }
[2026-09-29T09:03:31.720Z] an-a01n01 | SUCCESS => {
[2026-09-29T09:03:31.720Z]     "ansible_facts": {
[2026-09-29T09:03:31.720Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-29T09:03:31.720Z]     },
[2026-09-29T09:03:31.720Z]     "changed": false,
[2026-09-29T09:03:31.720Z]     "ping": "pong"
[2026-09-29T09:03:31.720Z] }
[2026-09-29T09:03:31.720Z] an-striker01 | SUCCESS => {
[2026-09-29T09:03:31.720Z]     "ansible_facts": {
[2026-09-29T09:03:31.720Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-29T09:03:31.720Z]     },
[2026-09-29T09:03:31.720Z]     "changed": false,
[2026-09-29T09:03:31.720Z]     "ping": "pong"
[2026-09-29T09:03:31.720Z] }
[2026-09-29T09:03:31.720Z] an-striker02 | SUCCESS => {
[2026-09-29T09:03:31.720Z]     "ansible_facts": {
[2026-09-29T09:03:31.720Z]         "discovered_interpreter_python": "/usr/bin/python3"
[2026-09-29T09:03:31.720Z]     },
[2026-09-29T09:03:31.720Z]     "changed": false,
[2026-09-29T09:03:31.720Z]     "ping": "pong"
[2026-09-29T09:03:31.720Z] }
[2026-09-29T09:03:31.720Z] 
[2026-09-29T09:03:32.592Z] 
[2026-09-29T09:03:32.592Z] PLAY [Setup all nodes] *********************************************************
[2026-09-29T09:03:32.592Z] 
[2026-09-29T09:03:32.592Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:03:35.256Z] ok: [an-a01n01]
[2026-09-29T09:03:35.256Z] ok: [an-a01n02]
[2026-09-29T09:03:35.256Z] ok: [an-striker01]
[2026-09-29T09:03:35.256Z] ok: [an-a01dr01]
[2026-09-29T09:03:35.256Z] ok: [an-striker02]
[2026-09-29T09:03:35.256Z] 
[2026-09-29T09:03:35.256Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] ***
[2026-09-29T09:03:36.128Z] changed: [an-a01dr01]
[2026-09-29T09:03:36.128Z] changed: [an-a01n01]
[2026-09-29T09:03:36.128Z] changed: [an-striker01]
[2026-09-29T09:03:36.128Z] changed: [an-striker02]
[2026-09-29T09:03:36.128Z] changed: [an-a01n02]
[2026-09-29T09:03:36.128Z] 
[2026-09-29T09:03:36.128Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] ***
[2026-09-29T09:03:36.999Z] ok: [an-a01dr01]
[2026-09-29T09:03:36.999Z] ok: [an-a01n01]
[2026-09-29T09:03:36.999Z] ok: [an-striker01]
[2026-09-29T09:03:36.999Z] ok: [an-a01n02]
[2026-09-29T09:03:36.999Z] ok: [an-striker02]
[2026-09-29T09:03:36.999Z] 
[2026-09-29T09:03:36.999Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] ***
[2026-09-29T09:03:38.394Z] changed: [an-a01dr01]
[2026-09-29T09:03:38.394Z] changed: [an-a01n01]
[2026-09-29T09:03:38.394Z] changed: [an-striker02]
[2026-09-29T09:03:38.394Z] changed: [an-striker01]
[2026-09-29T09:03:38.394Z] changed: [an-a01n02]
[2026-09-29T09:03:38.394Z] 
[2026-09-29T09:03:38.394Z] PLAY [setup strikers] **********************************************************
[2026-09-29T09:03:38.394Z] 
[2026-09-29T09:03:38.394Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:03:40.378Z] ok: [an-striker01]
[2026-09-29T09:03:40.378Z] ok: [an-striker02]
[2026-09-29T09:03:40.378Z] 
[2026-09-29T09:03:40.378Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] ***
[2026-09-29T09:03:42.362Z] changed: [an-striker01]
[2026-09-29T09:03:42.362Z] changed: [an-striker02]
[2026-09-29T09:03:42.362Z] 
[2026-09-29T09:03:42.362Z] PLAY [setup lvm storage] *******************************************************
[2026-09-29T09:03:42.362Z] 
[2026-09-29T09:03:42.362Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:03:45.810Z] ok: [an-a01dr01]
[2026-09-29T09:03:45.810Z] ok: [an-a01n01]
[2026-09-29T09:03:45.810Z] ok: [an-a01n02]
[2026-09-29T09:03:45.810Z] 
[2026-09-29T09:03:45.810Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg
[2026-09-29T09:03:45.810Z] ] ****************
[2026-09-29T09:03:46.682Z] ok: [an-a01dr01]
[2026-09-29T09:03:46.682Z] ok: [an-a01n02]
[2026-09-29T09:03:46.682Z] ok: [an-a01n01]
[2026-09-29T09:03:46.682Z] 
[2026-09-29T09:03:46.682Z] TASK [vgcheck _raw_params=end_play] ********************************************
[2026-09-29T09:03:46.682Z] skipping: [an-a01n01]
[2026-09-29T09:03:46.682Z] 
[2026-09-29T09:03:46.682Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] *********************
[2026-09-29T09:03:48.115Z] changed: [an-a01dr01]
[2026-09-29T09:03:48.115Z] changed: [an-a01n02]
[2026-09-29T09:03:48.115Z] changed: [an-a01n01]
[2026-09-29T09:03:48.115Z] 
[2026-09-29T09:03:48.115Z] PLAY [deploy anvil] ************************************************************
[2026-09-29T09:03:48.115Z] 
[2026-09-29T09:03:48.115Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:03:49.508Z] ok: [an-striker02]
[2026-09-29T09:03:49.508Z] ok: [an-striker01]
[2026-09-29T09:03:49.508Z] 
[2026-09-29T09:03:49.508Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] ***
[2026-09-29T09:03:50.379Z] ok: [an-striker02]
[2026-09-29T09:03:50.379Z] ok: [an-striker01]
[2026-09-29T09:03:50.379Z] 
[2026-09-29T09:03:50.379Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] ***
[2026-09-29T09:03:51.771Z] changed: [an-striker02]
[2026-09-29T09:03:51.771Z] changed: [an-striker01]
[2026-09-29T09:03:51.771Z] 
[2026-09-29T09:03:51.771Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg
[2026-09-29T09:03:51.771Z] ] ***
[2026-09-29T09:03:54.435Z] changed: [an-striker01]
[2026-09-29T09:03:54.435Z] changed: [an-striker02]
[2026-09-29T09:03:54.435Z] 
[2026-09-29T09:03:54.435Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:03:54.435Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:03:54.435Z]      --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';"
[2026-09-29T09:03:54.435Z] ] ***
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left).
[2026-09-29T09:09:52.583Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left).
[2026-09-29T09:09:52.583Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:09:52.583Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:09:52.583Z] 
[2026-09-29T09:09:52.583Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)"
[2026-09-29T09:09:52.583Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g')
[2026-09-29T09:09:52.583Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l)
[2026-09-29T09:09:52.583Z] if [ "$dbcon" = 1 ]  &&  [ "$dbhosts" = 2 ]; then
[2026-09-29T09:09:52.583Z]   echo "DB has completed sync"
[2026-09-29T09:09:52.583Z]   exit 0
[2026-09-29T09:09:52.583Z] fi
[2026-09-29T09:09:52.583Z] if [ "$dbcon" != 2 ]; then
[2026-09-29T09:09:52.583Z]   echo "Detected $dbcon db connections (expected 2)"
[2026-09-29T09:09:52.583Z]   exit 1
[2026-09-29T09:09:52.583Z] fi
[2026-09-29T09:09:52.583Z] ] ***
[2026-09-29T09:10:34.416Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left).
[2026-09-29T09:10:34.416Z] ok: [an-striker01]
[2026-09-29T09:10:34.416Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left).
[2026-09-29T09:10:34.416Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left).
[2026-09-29T09:10:34.416Z] ok: [an-striker02]
[2026-09-29T09:10:34.416Z] 
[2026-09-29T09:10:34.416Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:10:34.416Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:10:34.416Z]      --username=postgres anvil -t \
[2026-09-29T09:10:34.416Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \
[2026-09-29T09:10:34.416Z] grep altee | \
[2026-09-29T09:10:34.416Z] grep 1$ | \
[2026-09-29T09:10:34.416Z] wc -l
[2026-09-29T09:10:34.416Z] ] ***
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left).
[2026-09-29T09:16:17.388Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left).
[2026-09-29T09:20:14.131Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left).
[2026-09-29T09:20:14.131Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:20:14.131Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:20:14.131Z] 
[2026-09-29T09:20:14.131Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:20:14.131Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:20:14.131Z]      --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \
[2026-09-29T09:20:14.131Z] grep altee | \
[2026-09-29T09:20:14.131Z] wc -l
[2026-09-29T09:20:14.131Z] ] ***
[2026-09-29T09:20:14.131Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:20:14.131Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:20:14.131Z] 
[2026-09-29T09:20:14.131Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:20:14.131Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:20:14.131Z]      --username=postgres anvil -t \
[2026-09-29T09:20:14.131Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \
[2026-09-29T09:20:14.131Z] grep altee | \
[2026-09-29T09:20:14.131Z] grep 1$ | \
[2026-09-29T09:20:14.131Z] wc -l
[2026-09-29T09:20:14.131Z] ] ***
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left).
[2026-09-29T09:22:39.982Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left).
[2026-09-29T09:22:39.983Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (347 retries left).
[2026-09-29T09:22:39.983Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:22:39.983Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:22:39.983Z] 
[2026-09-29T09:22:39.983Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:22:39.983Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:22:39.983Z]      --username=postgres anvil -t \
[2026-09-29T09:22:39.983Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \
[2026-09-29T09:22:39.983Z] grep altee | \
[2026-09-29T09:22:39.983Z] grep 1$ | \
[2026-09-29T09:22:39.983Z] wc -l
[2026-09-29T09:22:39.983Z] ] ***
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left).
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left).
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left).
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left).
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left).
[2026-09-29T09:23:04.349Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (118 retries left).
[2026-09-29T09:23:04.349Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:23:04.349Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:23:04.349Z] 
[2026-09-29T09:23:04.349Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \
[2026-09-29T09:23:04.349Z] psql --host="{{ inventory_hostname }}" \
[2026-09-29T09:23:04.349Z]      --username=postgres anvil -t \
[2026-09-29T09:23:04.349Z]      -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \
[2026-09-29T09:23:04.349Z] grep altee | \
[2026-09-29T09:23:04.349Z] grep 1$ | \
[2026-09-29T09:23:04.349Z] wc -l
[2026-09-29T09:23:04.349Z] ] ***
[2026-09-29T09:23:04.349Z] ok: [an-striker01 -> localhost]
[2026-09-29T09:23:04.349Z] ok: [an-striker02 -> localhost]
[2026-09-29T09:23:04.349Z] 
[2026-09-29T09:23:04.349Z] PLAY [check network status after deployment] ***********************************
[2026-09-29T09:23:04.349Z] 
[2026-09-29T09:23:04.349Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:23:05.741Z] ok: [an-a01n02]
[2026-09-29T09:23:05.741Z] ok: [an-a01dr01]
[2026-09-29T09:23:05.741Z] ok: [an-a01n01]
[2026-09-29T09:23:05.741Z] ok: [an-striker01]
[2026-09-29T09:23:05.741Z] ok: [an-striker02]
[2026-09-29T09:23:05.741Z] 
[2026-09-29T09:23:05.741Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')"
[2026-09-29T09:23:05.741Z] if [ "$device" = "" ]; then
[2026-09-29T09:23:05.741Z]   echo "ERROR: unable to determine device for iface {{ item }}"
[2026-09-29T09:23:05.741Z]   exit 1
[2026-09-29T09:23:05.741Z] fi
[2026-09-29T09:23:05.741Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then
[2026-09-29T09:23:05.741Z]   echo "ERROR: {{ item }} / $device is not set to autoconnecct"
[2026-09-29T09:23:05.741Z]   exit 1
[2026-09-29T09:23:05.741Z] fi
[2026-09-29T09:23:05.741Z] exit 0
[2026-09-29T09:23:05.741Z] ] ***
[2026-09-29T09:23:05.741Z] skipping: [an-a01n01] => (item=lo) 
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=sn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=sn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=ifn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=bcn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=bcn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=bcn1_bridge1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=bcn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=bcn1_bridge1)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=ifn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=bcn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=bcn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=ifn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=sn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=ifn1_link1)
[2026-09-29T09:23:13.822Z] skipping: [an-striker01] => (item=lo) 
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=ifn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=ifn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=ifn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=ifn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=ifn1_bridge1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=bcn1_bridge1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=bcn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=ifn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=sn1_link2)
[2026-09-29T09:23:13.822Z] skipping: [an-a01dr01] => (item=lo) 
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=bcn1_link1)
[2026-09-29T09:23:13.822Z] skipping: [an-a01n02] => (item=lo) 
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=bcn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=bcn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=ifn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=bcn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=sn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=sn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=bcn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=ifn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=ifn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-striker01] => (item=bcn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=ifn1_link2)
[2026-09-29T09:23:13.822Z] skipping: [an-striker02] => (item=lo) 
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=sn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=ifn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=bcn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-striker02] => (item=bcn1_link1)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=ifn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=bcn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=sn1_link2)
[2026-09-29T09:23:13.822Z] changed: [an-a01dr01] => (item=sn1_bond1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n02] => (item=ifn1_bridge1)
[2026-09-29T09:23:13.822Z] changed: [an-a01n01] => (item=ifn1_bridge1)
[2026-09-29T09:23:13.822Z] 
[2026-09-29T09:23:13.822Z] PLAY [check pacemaker status after deployment] *********************************
[2026-09-29T09:23:13.822Z] 
[2026-09-29T09:23:13.822Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:23:16.000Z] ok: [an-a01n02]
[2026-09-29T09:23:16.000Z] ok: [an-a01n01]
[2026-09-29T09:23:16.000Z] 
[2026-09-29T09:23:16.000Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] ***
[2026-09-29T09:23:16.000Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T09:23:16.000Z] 
[2026-09-29T09:23:16.000Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T09:23:16.000Z] ] ***
[2026-09-29T09:23:19.446Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T09:23:19.446Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T09:23:19.446Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T09:23:19.446Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T09:23:19.446Z] 
[2026-09-29T09:23:19.446Z] PLAY [check drbd kmod] *********************************************************
[2026-09-29T09:23:19.446Z] 
[2026-09-29T09:23:19.446Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:23:21.431Z] ok: [an-a01dr01]
[2026-09-29T09:23:21.431Z] ok: [an-a01n01]
[2026-09-29T09:23:21.431Z] ok: [an-a01n02]
[2026-09-29T09:23:21.431Z] 
[2026-09-29T09:23:21.431Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd
[2026-09-29T09:23:21.431Z] ] ***
[2026-09-29T09:23:22.303Z] ok: [an-a01n01]
[2026-09-29T09:23:22.303Z] ok: [an-a01n02]
[2026-09-29T09:23:22.303Z] ok: [an-a01dr01]
[2026-09-29T09:23:22.303Z] 
[2026-09-29T09:23:22.303Z] TASK [load drbd kernel module _raw_params=modprobe drbd
[2026-09-29T09:23:22.303Z] ] ***********************
[2026-09-29T09:23:22.303Z] skipping: [an-a01n01]
[2026-09-29T09:23:22.303Z] skipping: [an-a01n02]
[2026-09-29T09:23:22.303Z] skipping: [an-a01dr01]
[2026-09-29T09:23:22.303Z] 
[2026-09-29T09:23:22.303Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd
[2026-09-29T09:23:22.303Z] ] ***
[2026-09-29T09:23:22.303Z] skipping: [an-a01n01]
[2026-09-29T09:23:22.303Z] skipping: [an-a01n02]
[2026-09-29T09:23:22.303Z] skipping: [an-a01dr01]
[2026-09-29T09:23:22.303Z] 
[2026-09-29T09:23:22.303Z] PLAY [Copy iso images to striker01] ********************************************
[2026-09-29T09:23:22.303Z] 
[2026-09-29T09:23:22.303Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:23:24.289Z] ok: [an-striker01]
[2026-09-29T09:23:24.290Z] 
[2026-09-29T09:23:24.290Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] *******************
[2026-09-29T09:23:25.161Z] ok: [an-striker01]
[2026-09-29T09:23:25.161Z] 
[2026-09-29T09:23:25.161Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ******
[2026-09-29T09:23:26.032Z] changed: [an-striker01]
[2026-09-29T09:23:26.032Z] 
[2026-09-29T09:23:26.032Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] ***
[2026-09-29T09:23:35.822Z] changed: [an-striker01] => (item=deploy1.iso)
[2026-09-29T09:23:35.822Z] changed: [an-striker01] => (item=deploy2.iso)
[2026-09-29T09:23:35.822Z] changed: [an-striker01] => (item=deploy3.iso)
[2026-09-29T09:23:35.822Z] changed: [an-striker01] => (item=deploy4.iso)
[2026-09-29T09:23:35.822Z] changed: [an-striker01] => (item=deploy5.iso)
[2026-09-29T09:23:35.822Z] 
[2026-09-29T09:23:35.822Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/
[2026-09-29T09:23:35.822Z] ] ***
[2026-09-29T09:23:35.822Z] changed: [an-striker01]
[2026-09-29T09:23:35.822Z] 
[2026-09-29T09:23:35.822Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] *******************
[2026-09-29T09:23:37.213Z] changed: [an-striker01]
[2026-09-29T09:23:37.213Z] 
[2026-09-29T09:23:37.213Z] PLAY [Check iso images propagation] ********************************************
[2026-09-29T09:23:37.213Z] 
[2026-09-29T09:23:37.213Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:23:39.390Z] ok: [an-a01dr01]
[2026-09-29T09:23:39.390Z] ok: [an-a01n02]
[2026-09-29T09:23:39.390Z] ok: [an-a01n01]
[2026-09-29T09:23:39.390Z] ok: [an-striker01]
[2026-09-29T09:23:39.390Z] ok: [an-striker02]
[2026-09-29T09:23:39.390Z] 
[2026-09-29T09:23:39.391Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then
[2026-09-29T09:23:39.391Z]  exit 1
[2026-09-29T09:23:39.391Z] else
[2026-09-29T09:23:39.391Z]  exit 0
[2026-09-29T09:23:39.391Z] fi
[2026-09-29T09:23:39.391Z] ] ***
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-29T09:25:19.485Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left).
[2026-09-29T09:25:19.485Z] ok: [an-a01dr01]
[2026-09-29T09:25:19.485Z] ok: [an-a01n02]
[2026-09-29T09:25:19.485Z] ok: [an-a01n01]
[2026-09-29T09:25:19.485Z] ok: [an-striker01]
[2026-09-29T09:25:19.485Z] ok: [an-striker02]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] PLAY [example tests] ***********************************************************
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:25:19.485Z] ok: [an-a01dr01]
[2026-09-29T09:25:19.485Z] ok: [an-a01n01]
[2026-09-29T09:25:19.485Z] ok: [an-a01n02]
[2026-09-29T09:25:19.485Z] ok: [an-striker02]
[2026-09-29T09:25:19.485Z] ok: [an-striker01]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true
[2026-09-29T09:25:19.485Z] ] ***
[2026-09-29T09:25:19.485Z] skipping: [an-a01n01]
[2026-09-29T09:25:19.485Z] skipping: [an-a01n02]
[2026-09-29T09:25:19.485Z] skipping: [an-a01dr01]
[2026-09-29T09:25:19.485Z] ok: [an-striker02]
[2026-09-29T09:25:19.485Z] ok: [an-striker01]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true
[2026-09-29T09:25:19.485Z] ] ***
[2026-09-29T09:25:19.485Z] skipping: [an-striker01]
[2026-09-29T09:25:19.485Z] skipping: [an-striker02]
[2026-09-29T09:25:19.485Z] ok: [an-a01n01]
[2026-09-29T09:25:19.485Z] ok: [an-a01n02]
[2026-09-29T09:25:19.485Z] ok: [an-a01dr01]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [inject failure _raw_params=/usr/bin/false
[2026-09-29T09:25:19.485Z] ] *******************************
[2026-09-29T09:25:19.485Z] skipping: [an-striker01]
[2026-09-29T09:25:19.485Z] skipping: [an-striker02]
[2026-09-29T09:25:19.485Z] skipping: [an-a01n01]
[2026-09-29T09:25:19.485Z] skipping: [an-a01n02]
[2026-09-29T09:25:19.485Z] skipping: [an-a01dr01]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] PLAY [Record anvil available resources from striker01] *************************
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:25:19.485Z] ok: [an-striker01]
[2026-09-29T09:25:19.485Z] 
[2026-09-29T09:25:19.485Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start
[2026-09-29T09:25:19.485Z] ] ***
[2026-09-29T09:25:19.896Z] ok: [an-striker01]
[2026-09-29T09:25:19.896Z] 
[2026-09-29T09:25:19.896Z] PLAY [check selinux status] ****************************************************
[2026-09-29T09:25:19.896Z] 
[2026-09-29T09:25:19.896Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:25:22.559Z] ok: [an-a01n01]
[2026-09-29T09:25:22.559Z] ok: [an-a01dr01]
[2026-09-29T09:25:22.559Z] ok: [an-a01n02]
[2026-09-29T09:25:22.559Z] ok: [an-striker02]
[2026-09-29T09:25:22.559Z] ok: [an-striker01]
[2026-09-29T09:25:22.559Z] 
[2026-09-29T09:25:22.559Z] TASK [check selinux status permissive (8) var=ansible_selinux] *****************
[2026-09-29T09:25:22.559Z] skipping: [an-striker01]
[2026-09-29T09:25:22.559Z] skipping: [an-striker02]
[2026-09-29T09:25:22.559Z] skipping: [an-a01n01]
[2026-09-29T09:25:22.559Z] skipping: [an-a01n02]
[2026-09-29T09:25:22.559Z] skipping: [an-a01dr01]
[2026-09-29T09:25:22.559Z] 
[2026-09-29T09:25:22.559Z] TASK [check selinux status permissive (9) var=ansible_selinux] *****************
[2026-09-29T09:25:22.559Z] ok: [an-striker01] => {
[2026-09-29T09:25:22.559Z]     "ansible_selinux": {
[2026-09-29T09:25:22.559Z]         "config_mode": "enforcing",
[2026-09-29T09:25:22.559Z]         "mode": "enforcing",
[2026-09-29T09:25:22.559Z]         "policyvers": 33,
[2026-09-29T09:25:22.559Z]         "status": "enabled",
[2026-09-29T09:25:22.559Z]         "type": "targeted"
[2026-09-29T09:25:22.559Z]     },
[2026-09-29T09:25:22.559Z]     "failed_when_result": false
[2026-09-29T09:25:22.559Z] }
[2026-09-29T09:25:22.970Z] ok: [an-striker02] => {
[2026-09-29T09:25:22.970Z]     "ansible_selinux": {
[2026-09-29T09:25:22.970Z]         "config_mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "policyvers": 33,
[2026-09-29T09:25:22.970Z]         "status": "enabled",
[2026-09-29T09:25:22.970Z]         "type": "targeted"
[2026-09-29T09:25:22.970Z]     },
[2026-09-29T09:25:22.970Z]     "failed_when_result": false
[2026-09-29T09:25:22.970Z] }
[2026-09-29T09:25:22.970Z] ok: [an-a01n01] => {
[2026-09-29T09:25:22.970Z]     "ansible_selinux": {
[2026-09-29T09:25:22.970Z]         "config_mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "policyvers": 33,
[2026-09-29T09:25:22.970Z]         "status": "enabled",
[2026-09-29T09:25:22.970Z]         "type": "targeted"
[2026-09-29T09:25:22.970Z]     },
[2026-09-29T09:25:22.970Z]     "failed_when_result": false
[2026-09-29T09:25:22.970Z] }
[2026-09-29T09:25:22.970Z] ok: [an-a01n02] => {
[2026-09-29T09:25:22.970Z]     "ansible_selinux": {
[2026-09-29T09:25:22.970Z]         "config_mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "policyvers": 33,
[2026-09-29T09:25:22.970Z]         "status": "enabled",
[2026-09-29T09:25:22.970Z]         "type": "targeted"
[2026-09-29T09:25:22.970Z]     },
[2026-09-29T09:25:22.970Z]     "failed_when_result": false
[2026-09-29T09:25:22.970Z] }
[2026-09-29T09:25:22.970Z] ok: [an-a01dr01] => {
[2026-09-29T09:25:22.970Z]     "ansible_selinux": {
[2026-09-29T09:25:22.970Z]         "config_mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "mode": "enforcing",
[2026-09-29T09:25:22.970Z]         "policyvers": 33,
[2026-09-29T09:25:22.970Z]         "status": "enabled",
[2026-09-29T09:25:22.970Z]         "type": "targeted"
[2026-09-29T09:25:22.970Z]     },
[2026-09-29T09:25:22.970Z]     "failed_when_result": false
[2026-09-29T09:25:22.970Z] }
[2026-09-29T09:25:22.970Z] 
[2026-09-29T09:25:22.970Z] PLAY [Create random file on striker01] *****************************************
[2026-09-29T09:25:22.970Z] 
[2026-09-29T09:25:22.970Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:25:25.674Z] ok: [an-striker01]
[2026-09-29T09:25:25.674Z] 
[2026-09-29T09:25:25.674Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] ***
[2026-09-29T09:25:27.659Z] changed: [an-striker01]
[2026-09-29T09:25:27.659Z] 
[2026-09-29T09:25:27.659Z] PLAY [Check archive file propagation] ******************************************
[2026-09-29T09:25:27.659Z] 
[2026-09-29T09:25:27.659Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:25:30.327Z] ok: [an-a01dr01]
[2026-09-29T09:25:30.327Z] ok: [an-a01n02]
[2026-09-29T09:25:30.327Z] ok: [an-striker02]
[2026-09-29T09:25:30.327Z] ok: [an-a01n01]
[2026-09-29T09:25:30.327Z] ok: [an-striker01]
[2026-09-29T09:25:30.327Z] 
[2026-09-29T09:25:30.327Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then
[2026-09-29T09:25:30.327Z]  exit 1
[2026-09-29T09:25:30.327Z] else
[2026-09-29T09:25:30.327Z]  exit 0
[2026-09-29T09:25:30.327Z] fi
[2026-09-29T09:25:30.327Z] ] ***
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left).
[2026-09-29T09:26:41.398Z] ok: [an-striker01]
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left).
[2026-09-29T09:26:41.398Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left).
[2026-09-29T09:26:41.398Z] ok: [an-a01dr01]
[2026-09-29T09:26:41.398Z] ok: [an-a01n01]
[2026-09-29T09:26:41.398Z] ok: [an-a01n02]
[2026-09-29T09:26:41.398Z] ok: [an-striker02]
[2026-09-29T09:26:41.398Z] 
[2026-09-29T09:26:41.398Z] PLAY [test fencing] ************************************************************
[2026-09-29T09:26:41.398Z] 
[2026-09-29T09:26:41.398Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:26:41.398Z] ok: [an-a01n02]
[2026-09-29T09:26:41.398Z] ok: [an-a01n01]
[2026-09-29T09:26:41.398Z] 
[2026-09-29T09:26:41.398Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T09:26:41.398Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T09:26:41.398Z] 
[2026-09-29T09:26:41.398Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T09:26:41.398Z] ] ***
[2026-09-29T09:26:44.063Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T09:26:44.063Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T09:26:44.063Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T09:26:44.063Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T09:26:44.063Z] 
[2026-09-29T09:26:44.063Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************
[2026-09-29T09:26:44.063Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T09:26:44.063Z] 
[2026-09-29T09:26:44.063Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T09:26:44.063Z] ] **********
[2026-09-29T09:26:44.933Z] skipping: [an-a01n02]
[2026-09-29T09:26:44.934Z] changed: [an-a01n01]
[2026-09-29T09:26:44.934Z] 
[2026-09-29T09:26:44.934Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T09:26:44.934Z] skipping: [an-a01n02]
[2026-09-29T09:26:44.934Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01
[2026-09-29T09:26:44.934Z] 
[2026-09-29T09:26:44.934Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T09:29:56.174Z] ok: [an-a01n01 -> localhost]
[2026-09-29T09:29:56.174Z] 
[2026-09-29T09:29:56.174Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T09:29:56.174Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T09:29:56.174Z] 
[2026-09-29T09:29:56.174Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T09:29:56.174Z] ] ***
[2026-09-29T09:30:00.524Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T09:30:00.524Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T09:30:00.524Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T09:30:00.524Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T09:30:00.524Z] 
[2026-09-29T09:30:00.524Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************
[2026-09-29T09:30:00.524Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T09:30:00.524Z] 
[2026-09-29T09:30:00.524Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T09:30:00.524Z] ] **********
[2026-09-29T09:30:00.934Z] skipping: [an-a01n01]
[2026-09-29T09:30:00.934Z] changed: [an-a01n02]
[2026-09-29T09:30:00.934Z] 
[2026-09-29T09:30:00.934Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T09:30:00.934Z] skipping: [an-a01n01]
[2026-09-29T09:30:00.934Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02
[2026-09-29T09:30:00.934Z] 
[2026-09-29T09:30:00.935Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T09:33:12.773Z] ok: [an-a01n02 -> localhost]
[2026-09-29T09:33:12.773Z] 
[2026-09-29T09:33:12.773Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T09:33:12.773Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T09:33:12.773Z] 
[2026-09-29T09:33:12.773Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T09:33:12.773Z] ] ***
[2026-09-29T09:33:16.318Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T09:33:16.318Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T09:33:16.318Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T09:33:16.318Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T09:33:16.318Z] 
[2026-09-29T09:33:16.318Z] PLAY [advanced server tests] ***************************************************
[2026-09-29T09:33:16.318Z] 
[2026-09-29T09:33:16.318Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T09:33:17.716Z] ok: [an-a01n01]
[2026-09-29T09:33:17.716Z] ok: [an-a01n02]
[2026-09-29T09:33:17.716Z] 
[2026-09-29T09:33:17.716Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T09:33:17.716Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T09:33:17.716Z] 
[2026-09-29T09:33:17.716Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T09:33:17.716Z] ] ***
[2026-09-29T09:33:22.181Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T09:33:22.181Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T09:33:22.181Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T09:33:22.181Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T09:33:22.181Z] 
[2026-09-29T09:33:22.181Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-29T09:33:22.181Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-29T09:33:22.181Z] 
[2026-09-29T09:33:22.181Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-29T09:33:22.181Z] if [ -z "$storage" ]; then
[2026-09-29T09:33:22.181Z]   echo "Unable to detect shared storage"
[2026-09-29T09:33:22.181Z]   exit 1
[2026-09-29T09:33:22.181Z] fi
[2026-09-29T09:33:22.181Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T09:33:22.181Z] ] ***
[2026-09-29T09:33:27.591Z] changed: [an-a01n01]
[2026-09-29T09:33:27.591Z] changed: [an-a01n02]
[2026-09-29T09:33:27.591Z] 
[2026-09-29T09:33:27.591Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-29T09:33:28.002Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-29T09:33:28.002Z] 
[2026-09-29T09:33:28.002Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-29T09:33:28.002Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-29T09:33:28.002Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-29T09:33:28.002Z]   echo "Unable to detect all deploy iso"
[2026-09-29T09:33:28.002Z]   echo "Detected isos"
[2026-09-29T09:33:28.002Z]   echo "$isos"
[2026-09-29T09:33:28.002Z]   exit 1
[2026-09-29T09:33:28.002Z] fi
[2026-09-29T09:33:28.002Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-29T09:33:28.002Z] if [ -z "$installiso" ]; then
[2026-09-29T09:33:28.002Z]   echo "Unable to detect install iso"
[2026-09-29T09:33:28.002Z]   echo "Detected isos"
[2026-09-29T09:33:28.002Z]   echo "$isos"
[2026-09-29T09:33:28.002Z]   exit 1
[2026-09-29T09:33:28.002Z] fi
[2026-09-29T09:33:28.002Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T09:33:28.002Z] ] ***
[2026-09-29T09:33:33.796Z] changed: [an-a01n01]
[2026-09-29T09:33:33.796Z] changed: [an-a01n02]
[2026-09-29T09:33:33.796Z] 
[2026-09-29T09:33:33.796Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] ***************
[2026-09-29T09:33:33.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T09:33:33.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T09:33:33.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T09:33:33.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T09:33:33.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T09:33:33.796Z] 
[2026-09-29T09:33:33.796Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T09:33:33.796Z] ] ***
[2026-09-29T09:33:35.189Z] skipping: [an-a01n02]
[2026-09-29T09:33:35.189Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.939125", "end": "2026-09-29 05:33:34.873048", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 05:33:33.933923", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T09:33:35.189Z] ...ignoring
[2026-09-29T09:33:35.189Z] 
[2026-09-29T09:33:35.189Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T09:33:35.189Z]   --ci-test \
[2026-09-29T09:33:35.189Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T09:33:35.189Z]   --os almalinux9 \
[2026-09-29T09:33:35.189Z]   --cpu 2 --ram 4G \
[2026-09-29T09:33:35.189Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T09:33:35.189Z]   --install-media {{ installisouuid }} \
[2026-09-29T09:33:35.189Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T09:33:35.189Z] ] ***
[2026-09-29T09:33:44.979Z] skipping: [an-a01n02]
[2026-09-29T09:33:44.979Z] changed: [an-a01n01]
[2026-09-29T09:33:44.979Z] 
[2026-09-29T09:33:44.979Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T09:33:44.979Z] skipping: [an-a01n01]
[2026-09-29T09:33:44.979Z] skipping: [an-a01n02]
[2026-09-29T09:33:44.979Z] 
[2026-09-29T09:33:44.979Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T09:33:44.979Z] skipping: [an-a01n01]
[2026-09-29T09:33:44.979Z] skipping: [an-a01n02]
[2026-09-29T09:33:44.979Z] 
[2026-09-29T09:33:44.979Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T09:33:44.979Z] skipping: [an-a01n01]
[2026-09-29T09:33:44.979Z] skipping: [an-a01n02]
[2026-09-29T09:33:44.979Z] 
[2026-09-29T09:33:44.979Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T09:33:44.979Z] skipping: [an-a01n01]
[2026-09-29T09:33:44.979Z] skipping: [an-a01n02]
[2026-09-29T09:33:44.979Z] 
[2026-09-29T09:33:44.979Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T09:33:44.979Z] ] ***
[2026-09-29T09:33:45.850Z] skipping: [an-a01n02]
[2026-09-29T09:33:45.850Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.964283", "end": "2026-09-29 05:33:45.515867", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 05:33:44.551584", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T09:33:45.850Z] ...ignoring
[2026-09-29T09:33:45.850Z] 
[2026-09-29T09:33:45.850Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T09:33:45.850Z]   --ci-test \
[2026-09-29T09:33:45.850Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T09:33:45.850Z]   --os almalinux9 \
[2026-09-29T09:33:45.850Z]   --cpu 2 --ram 4G \
[2026-09-29T09:33:45.850Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T09:33:45.850Z]   --install-media {{ installisouuid }} \
[2026-09-29T09:33:45.850Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T09:33:45.850Z] ] ***
[2026-09-29T09:33:55.639Z] skipping: [an-a01n02]
[2026-09-29T09:33:55.639Z] changed: [an-a01n01]
[2026-09-29T09:33:55.639Z] 
[2026-09-29T09:33:55.639Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T09:33:55.639Z] skipping: [an-a01n01]
[2026-09-29T09:33:55.639Z] skipping: [an-a01n02]
[2026-09-29T09:33:55.639Z] 
[2026-09-29T09:33:55.639Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T09:33:55.639Z] skipping: [an-a01n01]
[2026-09-29T09:33:55.639Z] skipping: [an-a01n02]
[2026-09-29T09:33:55.639Z] 
[2026-09-29T09:33:55.640Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T09:33:55.640Z] skipping: [an-a01n01]
[2026-09-29T09:33:55.640Z] skipping: [an-a01n02]
[2026-09-29T09:33:55.640Z] 
[2026-09-29T09:33:55.640Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T09:33:55.640Z] skipping: [an-a01n01]
[2026-09-29T09:33:55.640Z] skipping: [an-a01n02]
[2026-09-29T09:33:55.640Z] 
[2026-09-29T09:33:55.640Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T09:33:55.640Z] ] ***
[2026-09-29T09:33:57.032Z] skipping: [an-a01n02]
[2026-09-29T09:33:57.032Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.926560", "end": "2026-09-29 05:33:56.529071", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 05:33:55.602511", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T09:33:57.032Z] ...ignoring
[2026-09-29T09:33:57.032Z] 
[2026-09-29T09:33:57.032Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T09:33:57.032Z]   --ci-test \
[2026-09-29T09:33:57.032Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T09:33:57.032Z]   --os almalinux9 \
[2026-09-29T09:33:57.032Z]   --cpu 2 --ram 4G \
[2026-09-29T09:33:57.032Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T09:33:57.032Z]   --install-media {{ installisouuid }} \
[2026-09-29T09:33:57.032Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T09:33:57.032Z] ] ***
[2026-09-29T09:34:07.419Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.419Z] changed: [an-a01n01]
[2026-09-29T09:34:07.419Z] 
[2026-09-29T09:34:07.419Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T09:34:07.419Z] skipping: [an-a01n01]
[2026-09-29T09:34:07.419Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.419Z] 
[2026-09-29T09:34:07.419Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T09:34:07.419Z] skipping: [an-a01n01]
[2026-09-29T09:34:07.419Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.419Z] 
[2026-09-29T09:34:07.419Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T09:34:07.419Z] skipping: [an-a01n01]
[2026-09-29T09:34:07.419Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.419Z] 
[2026-09-29T09:34:07.419Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T09:34:07.419Z] skipping: [an-a01n01]
[2026-09-29T09:34:07.419Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.419Z] 
[2026-09-29T09:34:07.419Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T09:34:07.419Z] ] ***
[2026-09-29T09:34:07.830Z] skipping: [an-a01n02]
[2026-09-29T09:34:07.830Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.923425", "end": "2026-09-29 05:34:07.549106", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 05:34:06.625681", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T09:34:07.830Z] ...ignoring
[2026-09-29T09:34:07.830Z] 
[2026-09-29T09:34:07.830Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T09:34:07.830Z]   --ci-test \
[2026-09-29T09:34:07.830Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T09:34:07.830Z]   --os almalinux9 \
[2026-09-29T09:34:07.830Z]   --cpu 2 --ram 4G \
[2026-09-29T09:34:07.830Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T09:34:07.830Z]   --install-media {{ installisouuid }} \
[2026-09-29T09:34:07.830Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T09:34:07.830Z] ] ***
[2026-09-29T09:34:17.625Z] skipping: [an-a01n02]
[2026-09-29T09:34:17.625Z] changed: [an-a01n01]
[2026-09-29T09:34:17.625Z] 
[2026-09-29T09:34:17.625Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T09:34:17.625Z] skipping: [an-a01n01]
[2026-09-29T09:34:17.625Z] skipping: [an-a01n02]
[2026-09-29T09:34:17.625Z] 
[2026-09-29T09:34:17.625Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T09:34:17.625Z] skipping: [an-a01n01]
[2026-09-29T09:34:17.625Z] skipping: [an-a01n02]
[2026-09-29T09:34:17.625Z] 
[2026-09-29T09:34:17.625Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T09:34:17.625Z] skipping: [an-a01n01]
[2026-09-29T09:34:17.625Z] skipping: [an-a01n02]
[2026-09-29T09:34:17.625Z] 
[2026-09-29T09:34:17.625Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T09:34:17.625Z] skipping: [an-a01n01]
[2026-09-29T09:34:17.625Z] skipping: [an-a01n02]
[2026-09-29T09:34:17.625Z] 
[2026-09-29T09:34:17.625Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T09:34:17.625Z] ] ***
[2026-09-29T09:34:19.610Z] skipping: [an-a01n02]
[2026-09-29T09:34:19.610Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.029086", "end": "2026-09-29 05:34:18.999749", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 05:34:17.970663", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T09:34:19.610Z] ...ignoring
[2026-09-29T09:34:19.610Z] 
[2026-09-29T09:34:19.610Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T09:34:19.610Z]   --ci-test \
[2026-09-29T09:34:19.610Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T09:34:19.610Z]   --os almalinux9 \
[2026-09-29T09:34:19.610Z]   --cpu 2 --ram 4G \
[2026-09-29T09:34:19.610Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T09:34:19.610Z]   --install-media {{ installisouuid }} \
[2026-09-29T09:34:19.610Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T09:34:19.610Z] ] ***
[2026-09-29T09:34:32.596Z] skipping: [an-a01n02]
[2026-09-29T09:34:32.596Z] changed: [an-a01n01]
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T09:34:32.596Z] skipping: [an-a01n01]
[2026-09-29T09:34:32.596Z] skipping: [an-a01n02]
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T09:34:32.596Z] skipping: [an-a01n01]
[2026-09-29T09:34:32.596Z] skipping: [an-a01n02]
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T09:34:32.596Z] skipping: [an-a01n01]
[2026-09-29T09:34:32.596Z] skipping: [an-a01n02]
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T09:34:32.596Z] skipping: [an-a01n01]
[2026-09-29T09:34:32.596Z] skipping: [an-a01n02]
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] ***
[2026-09-29T09:34:32.596Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T09:34:32.596Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T09:34:32.596Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T09:34:32.596Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T09:34:32.596Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T09:34:32.596Z] 
[2026-09-29T09:34:32.596Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T09:34:32.596Z] ] ***
[2026-09-29T09:34:57.331Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left).
[2026-09-29T09:34:57.331Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left).
[2026-09-29T09:34:57.331Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left).
[2026-09-29T09:34:57.331Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left).
[2026-09-29T09:34:57.331Z] ok: [an-a01n02]
[2026-09-29T09:34:57.331Z] ok: [an-a01n01]
[2026-09-29T09:34:57.331Z] 
[2026-09-29T09:34:57.331Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T09:34:57.331Z] ] ***
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (348 retries left).
[2026-09-29T09:37:38.730Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (348 retries left).
[2026-09-29T09:37:38.730Z] ok: [an-a01n02]
[2026-09-29T09:37:38.730Z] ok: [an-a01n01]
[2026-09-29T09:37:38.730Z] 
[2026-09-29T09:37:38.730Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T09:37:38.730Z] ] ***
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left).
[2026-09-29T09:40:21.105Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left).
[2026-09-29T09:40:21.105Z] ok: [an-a01n02]
[2026-09-29T09:40:21.105Z] ok: [an-a01n01]
[2026-09-29T09:40:21.105Z] 
[2026-09-29T09:40:21.105Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T09:40:21.105Z] ] ***
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (347 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (346 retries left).
[2026-09-29T09:43:32.345Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (346 retries left).
[2026-09-29T09:43:32.345Z] ok: [an-a01n02]
[2026-09-29T09:43:32.345Z] ok: [an-a01n01]
[2026-09-29T09:43:32.345Z] 
[2026-09-29T09:43:32.345Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T09:43:32.345Z] ] ***
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (344 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left).
[2026-09-29T09:46:59.807Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (343 retries left).
[2026-09-29T09:46:59.807Z] ok: [an-a01n01]
[2026-09-29T09:46:59.807Z] ok: [an-a01n02]
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] ***
[2026-09-29T09:46:59.807Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T09:46:59.807Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T09:46:59.807Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T09:46:59.807Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T09:46:59.807Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T09:46:59.807Z] ] ***
[2026-09-29T09:46:59.807Z] ok: [an-a01n02]
[2026-09-29T09:46:59.807Z] ok: [an-a01n01]
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T09:46:59.807Z] ] ***
[2026-09-29T09:46:59.807Z] ok: [an-a01n02]
[2026-09-29T09:46:59.807Z] ok: [an-a01n01]
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T09:46:59.807Z] ] ***
[2026-09-29T09:46:59.807Z] ok: [an-a01n02]
[2026-09-29T09:46:59.807Z] ok: [an-a01n01]
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T09:46:59.807Z] ] ***
[2026-09-29T09:46:59.807Z] ok: [an-a01n02]
[2026-09-29T09:46:59.807Z] ok: [an-a01n01]
[2026-09-29T09:46:59.807Z] 
[2026-09-29T09:46:59.807Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T09:46:59.807Z] ] ***
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left).
[2026-09-29T09:48:24.743Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left).
[2026-09-29T09:48:24.743Z] ok: [an-a01n02]
[2026-09-29T09:48:24.743Z] ok: [an-a01n01]
[2026-09-29T09:48:24.743Z] 
[2026-09-29T09:48:24.743Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] ***
[2026-09-29T09:48:24.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=1)
[2026-09-29T09:48:24.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=2)
[2026-09-29T09:48:24.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=3)
[2026-09-29T09:48:24.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=4)
[2026-09-29T09:48:24.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=5)
[2026-09-29T09:48:24.743Z] 
[2026-09-29T09:48:24.743Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T09:48:24.743Z] ] ***
[2026-09-29T09:48:24.743Z] ok: [an-a01n02]
[2026-09-29T09:48:24.743Z] ok: [an-a01n01]
[2026-09-29T09:48:24.743Z] 
[2026-09-29T09:48:24.743Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T09:48:24.743Z] ] ***
[2026-09-29T09:48:24.743Z] ok: [an-a01n01]
[2026-09-29T09:48:24.743Z] ok: [an-a01n02]
[2026-09-29T09:48:24.743Z] 
[2026-09-29T09:48:24.743Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T09:48:24.743Z] ] ***
[2026-09-29T09:48:24.743Z] ok: [an-a01n02]
[2026-09-29T09:48:24.743Z] ok: [an-a01n01]
[2026-09-29T09:48:24.743Z] 
[2026-09-29T09:48:24.743Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T09:48:24.743Z] ] ***
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left).
[2026-09-29T09:51:36.275Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (347 retries left).
[2026-09-29T09:51:36.275Z] ok: [an-a01n01]
[2026-09-29T09:51:36.275Z] ok: [an-a01n02]
[2026-09-29T09:51:36.275Z] 
[2026-09-29T09:51:36.275Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T09:51:36.275Z] ] ***
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left).
[2026-09-29T09:55:48.418Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left).
[2026-09-29T09:55:48.418Z] ok: [an-a01n02]
[2026-09-29T09:55:48.418Z] ok: [an-a01n01]
[2026-09-29T09:55:48.418Z] 
[2026-09-29T09:55:48.418Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] ***
[2026-09-29T09:55:48.418Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T09:55:48.418Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T09:55:48.418Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T09:55:48.418Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T09:55:48.418Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T09:55:48.418Z] 
[2026-09-29T09:55:48.418Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T09:55:48.418Z] ] ***
[2026-09-29T09:55:48.418Z] ok: [an-a01n01]
[2026-09-29T09:55:48.418Z] ok: [an-a01n02]
[2026-09-29T09:55:48.418Z] 
[2026-09-29T09:55:48.418Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T09:55:48.418Z] ] ***
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left).
[2026-09-29T09:57:13.392Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left).
[2026-09-29T09:57:13.392Z] ok: [an-a01n01]
[2026-09-29T09:57:13.392Z] ok: [an-a01n02]
[2026-09-29T09:57:13.392Z] 
[2026-09-29T09:57:13.392Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T09:57:13.392Z] ] ***
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left).
[2026-09-29T09:58:24.421Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left).
[2026-09-29T09:58:24.421Z] ok: [an-a01n02]
[2026-09-29T09:58:24.421Z] ok: [an-a01n01]
[2026-09-29T09:58:24.421Z] 
[2026-09-29T09:58:24.421Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T09:58:24.421Z] ] ***
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (348 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (347 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (346 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (346 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (345 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (345 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (344 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (344 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (343 retries left).
[2026-09-29T10:01:50.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (343 retries left).
[2026-09-29T10:01:50.948Z] ok: [an-a01n01]
[2026-09-29T10:01:50.948Z] ok: [an-a01n02]
[2026-09-29T10:01:50.948Z] 
[2026-09-29T10:01:50.948Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:01:50.948Z] ] ***
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left).
[2026-09-29T10:05:02.363Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left).
[2026-09-29T10:05:02.363Z] ok: [an-a01n01]
[2026-09-29T10:05:02.363Z] ok: [an-a01n02]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] ***
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ******************
[2026-09-29T10:05:02.363Z] ok: [an-a01n01]
[2026-09-29T10:05:02.363Z] ok: [an-a01n02]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:05:02.363Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:05:02.363Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:05:02.363Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:05:02.363Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:05:02.363Z]   echo "pcs:"
[2026-09-29T10:05:02.363Z]   echo "$pcsstatus"
[2026-09-29T10:05:02.363Z]   exit 1
[2026-09-29T10:05:02.363Z] fi
[2026-09-29T10:05:02.363Z] exit 0
[2026-09-29T10:05:02.363Z] ] ***
[2026-09-29T10:05:02.363Z] ok: [an-a01n02]
[2026-09-29T10:05:02.363Z] ok: [an-a01n01]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:05:02.363Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:05:02.363Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:05:02.363Z] ret=0
[2026-09-29T10:05:02.363Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:05:02.363Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:05:02.363Z]   echo "$pcsstatus"
[2026-09-29T10:05:02.363Z]   ret=1
[2026-09-29T10:05:02.363Z] fi
[2026-09-29T10:05:02.363Z] echo "$hypervisor"
[2026-09-29T10:05:02.363Z] exit $ret
[2026-09-29T10:05:02.363Z] ] ***
[2026-09-29T10:05:02.363Z] ok: [an-a01n02]
[2026-09-29T10:05:02.363Z] ok: [an-a01n01]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-29T10:05:02.363Z] ok: [an-a01n01]
[2026-09-29T10:05:02.363Z] ok: [an-a01n02]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-29T10:05:02.363Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T10:05:02.363Z] ] **********
[2026-09-29T10:05:02.363Z] skipping: [an-a01n02]
[2026-09-29T10:05:02.363Z] changed: [an-a01n01]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T10:05:02.363Z] skipping: [an-a01n01]
[2026-09-29T10:05:02.363Z] skipping: [an-a01n02]
[2026-09-29T10:05:02.363Z] 
[2026-09-29T10:05:02.363Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-29T10:05:02.364Z] skipping: [an-a01n01]
[2026-09-29T10:05:02.364Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-29T10:05:02.364Z] 
[2026-09-29T10:05:02.364Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-29T10:05:02.364Z] ] ***
[2026-09-29T10:06:01.781Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-29T10:06:01.781Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-29T10:06:01.781Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-29T10:06:01.781Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-29T10:06:01.782Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-29T10:06:01.782Z] ok: [an-a01n02]
[2026-09-29T10:06:01.782Z] 
[2026-09-29T10:06:01.782Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:06:01.782Z] skipping: [an-a01n01]
[2026-09-29T10:06:01.782Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-29T10:06:01.782Z] 
[2026-09-29T10:06:01.782Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:06:01.782Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:06:01.782Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:06:01.782Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:06:01.782Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:06:01.782Z]   echo "pcs:"
[2026-09-29T10:06:01.782Z]   echo "$pcsstatus"
[2026-09-29T10:06:01.782Z]   exit 1
[2026-09-29T10:06:01.782Z] fi
[2026-09-29T10:06:01.782Z] exit 0
[2026-09-29T10:06:01.782Z] ] ***
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left).
[2026-09-29T10:08:27.769Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (112 retries left).
[2026-09-29T10:08:27.770Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (111 retries left).
[2026-09-29T10:08:27.770Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (110 retries left).
[2026-09-29T10:08:27.770Z] ok: [an-a01n02]
[2026-09-29T10:08:27.770Z] 
[2026-09-29T10:08:27.770Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:08:27.770Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:08:27.770Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:08:27.770Z] ret=0
[2026-09-29T10:08:27.770Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:08:27.770Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:08:27.770Z]   echo "$pcsstatus"
[2026-09-29T10:08:27.770Z]   ret=1
[2026-09-29T10:08:27.770Z] fi
[2026-09-29T10:08:27.770Z] echo "$hypervisor"
[2026-09-29T10:08:27.770Z] exit $ret
[2026-09-29T10:08:27.770Z] ] ***
[2026-09-29T10:08:27.770Z] ok: [an-a01n02]
[2026-09-29T10:08:27.770Z] 
[2026-09-29T10:08:27.770Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01] => (item=1) 
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01] => (item=2) 
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01] => (item=3) 
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01] => (item=4) 
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01] => (item=5) 
[2026-09-29T10:08:27.770Z] skipping: [an-a01n01]
[2026-09-29T10:08:27.770Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-29T10:08:27.770Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-29T10:08:27.770Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-29T10:08:27.770Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-29T10:08:27.770Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-29T10:08:27.770Z] 
[2026-09-29T10:08:27.770Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:08:27.770Z] ] ***
[2026-09-29T10:08:43.532Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-29T10:08:43.532Z] ok: [an-a01n02]
[2026-09-29T10:08:43.532Z] 
[2026-09-29T10:08:43.532Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:08:43.532Z] ] ***
[2026-09-29T10:08:43.532Z] ok: [an-a01n02]
[2026-09-29T10:08:43.532Z] 
[2026-09-29T10:08:43.532Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:08:43.532Z] ] ***
[2026-09-29T10:08:43.532Z] ok: [an-a01n02]
[2026-09-29T10:08:43.532Z] 
[2026-09-29T10:08:43.532Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:08:43.532Z] ] ***
[2026-09-29T10:08:43.943Z] ok: [an-a01n02]
[2026-09-29T10:08:43.943Z] 
[2026-09-29T10:08:43.943Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:08:43.943Z] ] ***
[2026-09-29T10:08:44.813Z] ok: [an-a01n02]
[2026-09-29T10:08:44.813Z] 
[2026-09-29T10:08:44.813Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-29T10:08:44.813Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:08:44.813Z] 
[2026-09-29T10:08:44.813Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T10:11:56.324Z] ok: [an-a01n01 -> localhost]
[2026-09-29T10:11:56.324Z] ok: [an-a01n02 -> localhost]
[2026-09-29T10:11:56.324Z] 
[2026-09-29T10:11:56.324Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-29T10:11:56.324Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:11:56.324Z] 
[2026-09-29T10:11:56.324Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:11:56.324Z] ] ***
[2026-09-29T10:12:01.188Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:12:01.188Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:12:01.188Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:12:01.188Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:12:01.188Z] 
[2026-09-29T10:12:01.188Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:12:01.188Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:12:01.188Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:12:01.188Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:12:01.188Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:12:01.188Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:12:01.188Z] 
[2026-09-29T10:12:01.188Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:12:01.188Z] ] ***
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-29T10:12:51.342Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left).
[2026-09-29T10:12:51.342Z] ok: [an-a01n02]
[2026-09-29T10:12:51.342Z] ok: [an-a01n01]
[2026-09-29T10:12:51.342Z] 
[2026-09-29T10:12:51.342Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:12:51.342Z] ] ***
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (351 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (351 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (350 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (350 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (349 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (349 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (348 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (348 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (347 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (347 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (346 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (346 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (345 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (345 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (344 retries left).
[2026-09-29T10:15:48.709Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (344 retries left).
[2026-09-29T10:15:48.709Z] ok: [an-a01n02]
[2026-09-29T10:15:48.709Z] ok: [an-a01n01]
[2026-09-29T10:15:48.709Z] 
[2026-09-29T10:15:48.709Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:15:48.709Z] ] ***
[2026-09-29T10:15:48.709Z] ok: [an-a01n02]
[2026-09-29T10:15:48.709Z] ok: [an-a01n01]
[2026-09-29T10:15:48.709Z] 
[2026-09-29T10:15:48.709Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:15:48.709Z] ] ***
[2026-09-29T10:15:48.709Z] ok: [an-a01n01]
[2026-09-29T10:15:48.709Z] ok: [an-a01n02]
[2026-09-29T10:15:48.709Z] 
[2026-09-29T10:15:48.709Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:15:48.709Z] ] ***
[2026-09-29T10:15:49.580Z] ok: [an-a01n01]
[2026-09-29T10:15:49.580Z] ok: [an-a01n02]
[2026-09-29T10:15:49.580Z] 
[2026-09-29T10:15:49.580Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ******************
[2026-09-29T10:15:49.580Z] ok: [an-a01n01]
[2026-09-29T10:15:49.580Z] ok: [an-a01n02]
[2026-09-29T10:15:49.580Z] 
[2026-09-29T10:15:49.580Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:15:49.580Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:15:49.580Z] 
[2026-09-29T10:15:49.580Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:15:49.580Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:15:49.580Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:15:49.580Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:15:49.580Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:15:49.580Z]   echo "pcs:"
[2026-09-29T10:15:49.580Z]   echo "$pcsstatus"
[2026-09-29T10:15:49.580Z]   exit 1
[2026-09-29T10:15:49.580Z] fi
[2026-09-29T10:15:49.580Z] exit 0
[2026-09-29T10:15:49.580Z] ] ***
[2026-09-29T10:15:51.566Z] ok: [an-a01n02]
[2026-09-29T10:15:51.566Z] ok: [an-a01n01]
[2026-09-29T10:15:51.566Z] 
[2026-09-29T10:15:51.566Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:15:51.566Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:15:51.566Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:15:51.566Z] ret=0
[2026-09-29T10:15:51.566Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:15:51.566Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:15:51.566Z]   echo "$pcsstatus"
[2026-09-29T10:15:51.566Z]   ret=1
[2026-09-29T10:15:51.566Z] fi
[2026-09-29T10:15:51.566Z] echo "$hypervisor"
[2026-09-29T10:15:51.566Z] exit $ret
[2026-09-29T10:15:51.566Z] ] ***
[2026-09-29T10:15:54.381Z] ok: [an-a01n02]
[2026-09-29T10:15:54.381Z] ok: [an-a01n01]
[2026-09-29T10:15:54.381Z] 
[2026-09-29T10:15:54.381Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-29T10:15:54.381Z] ok: [an-a01n01]
[2026-09-29T10:15:54.381Z] ok: [an-a01n02]
[2026-09-29T10:15:54.381Z] 
[2026-09-29T10:15:54.381Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-29T10:15:54.381Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:15:54.381Z] 
[2026-09-29T10:15:54.381Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T10:15:54.381Z] ] **********
[2026-09-29T10:15:54.793Z] skipping: [an-a01n01]
[2026-09-29T10:15:54.793Z] changed: [an-a01n02]
[2026-09-29T10:15:54.793Z] 
[2026-09-29T10:15:54.793Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T10:15:54.793Z] skipping: [an-a01n01]
[2026-09-29T10:15:54.793Z] skipping: [an-a01n02]
[2026-09-29T10:15:54.793Z] 
[2026-09-29T10:15:54.793Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-29T10:15:54.793Z] skipping: [an-a01n02]
[2026-09-29T10:15:54.793Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01
[2026-09-29T10:15:54.793Z] 
[2026-09-29T10:15:54.793Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-29T10:15:54.793Z] ] ***
[2026-09-29T10:17:05.852Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left).
[2026-09-29T10:17:05.852Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left).
[2026-09-29T10:17:05.852Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left).
[2026-09-29T10:17:05.852Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left).
[2026-09-29T10:17:05.852Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left).
[2026-09-29T10:17:05.852Z] ok: [an-a01n01]
[2026-09-29T10:17:05.852Z] 
[2026-09-29T10:17:05.852Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:17:05.852Z] skipping: [an-a01n02]
[2026-09-29T10:17:05.852Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01
[2026-09-29T10:17:05.852Z] 
[2026-09-29T10:17:05.852Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:17:05.852Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:17:05.852Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:17:05.852Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:17:05.852Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:17:05.852Z]   echo "pcs:"
[2026-09-29T10:17:05.852Z]   echo "$pcsstatus"
[2026-09-29T10:17:05.852Z]   exit 1
[2026-09-29T10:17:05.852Z] fi
[2026-09-29T10:17:05.852Z] exit 0
[2026-09-29T10:17:05.852Z] ] ***
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-29T10:18:18.509Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-29T10:18:18.509Z] ok: [an-a01n01]
[2026-09-29T10:18:18.509Z] 
[2026-09-29T10:18:18.509Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:18:18.509Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:18:18.509Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:18:18.509Z] ret=0
[2026-09-29T10:18:18.509Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:18:18.509Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:18:18.509Z]   echo "$pcsstatus"
[2026-09-29T10:18:18.509Z]   ret=1
[2026-09-29T10:18:18.509Z] fi
[2026-09-29T10:18:18.509Z] echo "$hypervisor"
[2026-09-29T10:18:18.509Z] exit $ret
[2026-09-29T10:18:18.509Z] ] ***
[2026-09-29T10:18:18.509Z] ok: [an-a01n01]
[2026-09-29T10:18:18.509Z] 
[2026-09-29T10:18:18.509Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02] => (item=1) 
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02] => (item=2) 
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02] => (item=3) 
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02] => (item=4) 
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02] => (item=5) 
[2026-09-29T10:18:18.509Z] skipping: [an-a01n02]
[2026-09-29T10:18:18.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1)
[2026-09-29T10:18:18.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2)
[2026-09-29T10:18:18.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3)
[2026-09-29T10:18:18.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4)
[2026-09-29T10:18:18.509Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5)
[2026-09-29T10:18:18.509Z] 
[2026-09-29T10:18:18.509Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:18:18.509Z] ] ***
[2026-09-29T10:18:30.308Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-29T10:18:30.308Z] ok: [an-a01n01]
[2026-09-29T10:18:30.308Z] 
[2026-09-29T10:18:30.308Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:18:30.308Z] ] ***
[2026-09-29T10:18:30.719Z] ok: [an-a01n01]
[2026-09-29T10:18:30.719Z] 
[2026-09-29T10:18:30.719Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:18:30.719Z] ] ***
[2026-09-29T10:18:31.591Z] ok: [an-a01n01]
[2026-09-29T10:18:31.591Z] 
[2026-09-29T10:18:31.591Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:18:31.591Z] ] ***
[2026-09-29T10:18:32.002Z] ok: [an-a01n01]
[2026-09-29T10:18:32.002Z] 
[2026-09-29T10:18:32.002Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:18:32.002Z] ] ***
[2026-09-29T10:18:32.873Z] ok: [an-a01n01]
[2026-09-29T10:18:32.873Z] 
[2026-09-29T10:18:32.873Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-29T10:18:32.873Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:18:32.873Z] 
[2026-09-29T10:18:32.873Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T10:21:44.741Z] ok: [an-a01n01 -> localhost]
[2026-09-29T10:21:44.741Z] ok: [an-a01n02 -> localhost]
[2026-09-29T10:21:44.741Z] 
[2026-09-29T10:21:44.741Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-29T10:21:44.741Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:21:44.741Z] 
[2026-09-29T10:21:44.741Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:21:44.741Z] ] ***
[2026-09-29T10:21:49.093Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:21:49.093Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:21:49.093Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:21:49.093Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:21:49.093Z] 
[2026-09-29T10:21:49.093Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:21:49.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:21:49.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:21:49.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:21:49.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:21:49.093Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:21:49.093Z] 
[2026-09-29T10:21:49.093Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:21:49.093Z] ] ***
[2026-09-29T10:21:50.486Z] ok: [an-a01n01]
[2026-09-29T10:21:50.486Z] ok: [an-a01n02]
[2026-09-29T10:21:50.486Z] 
[2026-09-29T10:21:50.486Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:21:50.486Z] ] ***
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (351 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (351 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (350 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (350 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (349 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (349 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (348 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (348 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (347 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (347 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (346 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (346 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (345 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (345 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (344 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (344 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (343 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (343 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (342 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (342 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (341 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (341 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (340 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (340 retries left).
[2026-09-29T10:25:47.503Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (339 retries left).
[2026-09-29T10:25:47.503Z] ok: [an-a01n02]
[2026-09-29T10:25:47.503Z] ok: [an-a01n01]
[2026-09-29T10:25:47.503Z] 
[2026-09-29T10:25:47.503Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:25:47.503Z] ] ***
[2026-09-29T10:25:47.503Z] ok: [an-a01n02]
[2026-09-29T10:25:47.503Z] ok: [an-a01n01]
[2026-09-29T10:25:47.503Z] 
[2026-09-29T10:25:47.503Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:25:47.503Z] ] ***
[2026-09-29T10:25:47.503Z] ok: [an-a01n01]
[2026-09-29T10:25:47.503Z] ok: [an-a01n02]
[2026-09-29T10:25:47.503Z] 
[2026-09-29T10:25:47.503Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:25:47.503Z] ] ***
[2026-09-29T10:25:48.724Z] ok: [an-a01n01]
[2026-09-29T10:25:48.724Z] ok: [an-a01n02]
[2026-09-29T10:25:48.724Z] 
[2026-09-29T10:25:48.724Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ******************
[2026-09-29T10:25:48.724Z] ok: [an-a01n01]
[2026-09-29T10:25:48.724Z] ok: [an-a01n02]
[2026-09-29T10:25:48.724Z] 
[2026-09-29T10:25:48.724Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:25:48.724Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:25:48.724Z] 
[2026-09-29T10:25:48.724Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:25:48.724Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:25:48.724Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:25:48.724Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:25:48.724Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:25:48.724Z]   echo "pcs:"
[2026-09-29T10:25:48.724Z]   echo "$pcsstatus"
[2026-09-29T10:25:48.724Z]   exit 1
[2026-09-29T10:25:48.724Z] fi
[2026-09-29T10:25:48.724Z] exit 0
[2026-09-29T10:25:48.724Z] ] ***
[2026-09-29T10:25:50.787Z] ok: [an-a01n02]
[2026-09-29T10:25:50.787Z] ok: [an-a01n01]
[2026-09-29T10:25:50.787Z] 
[2026-09-29T10:25:50.787Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:25:50.787Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:25:50.787Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:25:50.787Z] ret=0
[2026-09-29T10:25:50.787Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:25:50.787Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:25:50.787Z]   echo "$pcsstatus"
[2026-09-29T10:25:50.787Z]   ret=1
[2026-09-29T10:25:50.787Z] fi
[2026-09-29T10:25:50.787Z] echo "$hypervisor"
[2026-09-29T10:25:50.787Z] exit $ret
[2026-09-29T10:25:50.787Z] ] ***
[2026-09-29T10:25:52.771Z] ok: [an-a01n01]
[2026-09-29T10:25:52.771Z] ok: [an-a01n02]
[2026-09-29T10:25:52.771Z] 
[2026-09-29T10:25:52.771Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-29T10:25:52.771Z] ok: [an-a01n01]
[2026-09-29T10:25:52.771Z] ok: [an-a01n02]
[2026-09-29T10:25:52.771Z] 
[2026-09-29T10:25:52.771Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-29T10:25:52.771Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:25:52.771Z] 
[2026-09-29T10:25:52.771Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T10:25:52.771Z] ] **********
[2026-09-29T10:25:53.643Z] skipping: [an-a01n02]
[2026-09-29T10:25:53.643Z] changed: [an-a01n01]
[2026-09-29T10:25:53.643Z] 
[2026-09-29T10:25:53.643Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T10:25:53.643Z] skipping: [an-a01n01]
[2026-09-29T10:25:53.643Z] skipping: [an-a01n02]
[2026-09-29T10:25:53.643Z] 
[2026-09-29T10:25:53.643Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-29T10:25:53.643Z] skipping: [an-a01n01]
[2026-09-29T10:25:53.643Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-29T10:25:53.643Z] 
[2026-09-29T10:25:53.643Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-29T10:25:53.643Z] ] ***
[2026-09-29T10:27:04.674Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-29T10:27:04.674Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-29T10:27:04.674Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-29T10:27:04.674Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-29T10:27:04.674Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-29T10:27:04.674Z] ok: [an-a01n02]
[2026-09-29T10:27:04.674Z] 
[2026-09-29T10:27:04.674Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:27:04.674Z] skipping: [an-a01n01]
[2026-09-29T10:27:04.674Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-29T10:27:04.674Z] 
[2026-09-29T10:27:04.674Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:27:04.674Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:27:04.674Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:27:04.674Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:27:04.674Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:27:04.674Z]   echo "pcs:"
[2026-09-29T10:27:04.674Z]   echo "$pcsstatus"
[2026-09-29T10:27:04.674Z]   exit 1
[2026-09-29T10:27:04.674Z] fi
[2026-09-29T10:27:04.674Z] exit 0
[2026-09-29T10:27:04.674Z] ] ***
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-29T10:28:16.134Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left).
[2026-09-29T10:28:16.134Z] ok: [an-a01n02]
[2026-09-29T10:28:16.134Z] 
[2026-09-29T10:28:16.134Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:28:16.134Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:28:16.134Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:28:16.134Z] ret=0
[2026-09-29T10:28:16.134Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:28:16.134Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:28:16.134Z]   echo "$pcsstatus"
[2026-09-29T10:28:16.134Z]   ret=1
[2026-09-29T10:28:16.134Z] fi
[2026-09-29T10:28:16.134Z] echo "$hypervisor"
[2026-09-29T10:28:16.134Z] exit $ret
[2026-09-29T10:28:16.134Z] ] ***
[2026-09-29T10:28:16.134Z] ok: [an-a01n02]
[2026-09-29T10:28:16.134Z] 
[2026-09-29T10:28:16.134Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01] => (item=1) 
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01] => (item=2) 
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01] => (item=3) 
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01] => (item=4) 
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01] => (item=5) 
[2026-09-29T10:28:16.134Z] skipping: [an-a01n01]
[2026-09-29T10:28:16.134Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-29T10:28:16.134Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-29T10:28:16.134Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-29T10:28:16.134Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-29T10:28:16.134Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-29T10:28:16.134Z] 
[2026-09-29T10:28:16.134Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:28:16.134Z] ] ***
[2026-09-29T10:28:16.134Z] ok: [an-a01n02]
[2026-09-29T10:28:16.134Z] 
[2026-09-29T10:28:16.134Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:28:16.134Z] ] ***
[2026-09-29T10:28:18.798Z] ok: [an-a01n02]
[2026-09-29T10:28:18.798Z] 
[2026-09-29T10:28:18.798Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:28:18.798Z] ] ***
[2026-09-29T10:28:19.669Z] ok: [an-a01n02]
[2026-09-29T10:28:19.669Z] 
[2026-09-29T10:28:19.669Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:28:19.669Z] ] ***
[2026-09-29T10:28:21.060Z] ok: [an-a01n02]
[2026-09-29T10:28:21.060Z] 
[2026-09-29T10:28:21.060Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:28:21.060Z] ] ***
[2026-09-29T10:28:21.931Z] ok: [an-a01n02]
[2026-09-29T10:28:21.931Z] 
[2026-09-29T10:28:21.931Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-29T10:28:21.931Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:28:21.931Z] 
[2026-09-29T10:28:21.931Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T10:31:33.337Z] ok: [an-a01n02 -> localhost]
[2026-09-29T10:31:33.337Z] ok: [an-a01n01 -> localhost]
[2026-09-29T10:31:33.337Z] 
[2026-09-29T10:31:33.337Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-29T10:31:33.337Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:31:33.337Z] 
[2026-09-29T10:31:33.337Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:31:33.337Z] ] ***
[2026-09-29T10:31:37.690Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:31:37.690Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:31:37.690Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:31:37.690Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:31:37.690Z] 
[2026-09-29T10:31:37.690Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:31:37.690Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:31:37.690Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:31:37.690Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:31:37.690Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:31:37.690Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:31:37.690Z] 
[2026-09-29T10:31:37.690Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:31:37.690Z] ] ***
[2026-09-29T10:31:38.101Z] ok: [an-a01n02]
[2026-09-29T10:31:38.101Z] ok: [an-a01n01]
[2026-09-29T10:31:38.101Z] 
[2026-09-29T10:31:38.101Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:31:38.101Z] ] ***
[2026-09-29T10:31:38.748Z] ok: [an-a01n01]
[2026-09-29T10:31:38.748Z] ok: [an-a01n02]
[2026-09-29T10:31:38.748Z] 
[2026-09-29T10:31:38.748Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:31:38.748Z] ] ***
[2026-09-29T10:31:39.619Z] ok: [an-a01n02]
[2026-09-29T10:31:39.619Z] ok: [an-a01n01]
[2026-09-29T10:31:39.619Z] 
[2026-09-29T10:31:39.619Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:31:39.619Z] ] ***
[2026-09-29T10:31:40.490Z] ok: [an-a01n02]
[2026-09-29T10:31:40.490Z] ok: [an-a01n01]
[2026-09-29T10:31:40.490Z] 
[2026-09-29T10:31:40.490Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:31:40.490Z] ] ***
[2026-09-29T10:31:40.900Z] ok: [an-a01n02]
[2026-09-29T10:31:40.900Z] ok: [an-a01n01]
[2026-09-29T10:31:40.900Z] 
[2026-09-29T10:31:40.900Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ******************
[2026-09-29T10:31:40.900Z] ok: [an-a01n01]
[2026-09-29T10:31:40.900Z] ok: [an-a01n02]
[2026-09-29T10:31:40.900Z] 
[2026-09-29T10:31:40.900Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:31:41.311Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:31:41.311Z] 
[2026-09-29T10:31:41.311Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:31:41.311Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:31:41.311Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:31:41.311Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:31:41.311Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:31:41.311Z]   echo "pcs:"
[2026-09-29T10:31:41.311Z]   echo "$pcsstatus"
[2026-09-29T10:31:41.311Z]   exit 1
[2026-09-29T10:31:41.311Z] fi
[2026-09-29T10:31:41.311Z] exit 0
[2026-09-29T10:31:41.311Z] ] ***
[2026-09-29T10:31:43.296Z] ok: [an-a01n02]
[2026-09-29T10:31:43.296Z] ok: [an-a01n01]
[2026-09-29T10:31:43.296Z] 
[2026-09-29T10:31:43.296Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:31:43.296Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:31:43.296Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:31:43.296Z] ret=0
[2026-09-29T10:31:43.296Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:31:43.296Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:31:43.296Z]   echo "$pcsstatus"
[2026-09-29T10:31:43.296Z]   ret=1
[2026-09-29T10:31:43.296Z] fi
[2026-09-29T10:31:43.296Z] echo "$hypervisor"
[2026-09-29T10:31:43.296Z] exit $ret
[2026-09-29T10:31:43.296Z] ] ***
[2026-09-29T10:31:45.280Z] ok: [an-a01n02]
[2026-09-29T10:31:45.280Z] ok: [an-a01n01]
[2026-09-29T10:31:45.280Z] 
[2026-09-29T10:31:45.280Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-29T10:31:45.280Z] ok: [an-a01n01]
[2026-09-29T10:31:45.280Z] ok: [an-a01n02]
[2026-09-29T10:31:45.280Z] 
[2026-09-29T10:31:45.280Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-29T10:31:45.280Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:31:45.280Z] 
[2026-09-29T10:31:45.280Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T10:31:45.280Z] ] **********
[2026-09-29T10:31:46.151Z] skipping: [an-a01n01]
[2026-09-29T10:31:46.151Z] changed: [an-a01n02]
[2026-09-29T10:31:46.151Z] 
[2026-09-29T10:31:46.151Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T10:31:46.151Z] skipping: [an-a01n01]
[2026-09-29T10:31:46.151Z] skipping: [an-a01n02]
[2026-09-29T10:31:46.151Z] 
[2026-09-29T10:31:46.151Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-29T10:31:46.151Z] skipping: [an-a01n02]
[2026-09-29T10:31:46.151Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01
[2026-09-29T10:31:46.151Z] 
[2026-09-29T10:31:46.151Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-29T10:31:46.151Z] ] ***
[2026-09-29T10:32:57.459Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left).
[2026-09-29T10:32:57.459Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left).
[2026-09-29T10:32:57.459Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left).
[2026-09-29T10:32:57.459Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left).
[2026-09-29T10:32:57.459Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left).
[2026-09-29T10:32:57.459Z] ok: [an-a01n01]
[2026-09-29T10:32:57.459Z] 
[2026-09-29T10:32:57.459Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:32:57.459Z] skipping: [an-a01n02]
[2026-09-29T10:32:57.459Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01
[2026-09-29T10:32:57.459Z] 
[2026-09-29T10:32:57.459Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:32:57.459Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:32:57.459Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:32:57.459Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:32:57.459Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:32:57.459Z]   echo "pcs:"
[2026-09-29T10:32:57.459Z]   echo "$pcsstatus"
[2026-09-29T10:32:57.459Z]   exit 1
[2026-09-29T10:32:57.459Z] fi
[2026-09-29T10:32:57.459Z] exit 0
[2026-09-29T10:32:57.459Z] ] ***
[2026-09-29T10:33:56.882Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-29T10:33:56.882Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-29T10:33:56.882Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-29T10:33:56.882Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-29T10:33:56.882Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-29T10:33:56.882Z] ok: [an-a01n01]
[2026-09-29T10:33:56.882Z] 
[2026-09-29T10:33:56.882Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:33:56.882Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:33:56.882Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:33:56.882Z] ret=0
[2026-09-29T10:33:56.882Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:33:56.882Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:33:56.882Z]   echo "$pcsstatus"
[2026-09-29T10:33:56.882Z]   ret=1
[2026-09-29T10:33:56.882Z] fi
[2026-09-29T10:33:56.882Z] echo "$hypervisor"
[2026-09-29T10:33:56.882Z] exit $ret
[2026-09-29T10:33:56.882Z] ] ***
[2026-09-29T10:33:56.882Z] ok: [an-a01n01]
[2026-09-29T10:33:56.882Z] 
[2026-09-29T10:33:56.882Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02] => (item=1) 
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02] => (item=2) 
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02] => (item=3) 
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02] => (item=4) 
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02] => (item=5) 
[2026-09-29T10:33:56.882Z] skipping: [an-a01n02]
[2026-09-29T10:33:56.882Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1)
[2026-09-29T10:33:56.882Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2)
[2026-09-29T10:33:56.882Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3)
[2026-09-29T10:33:56.882Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4)
[2026-09-29T10:33:56.882Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5)
[2026-09-29T10:33:56.882Z] 
[2026-09-29T10:33:56.882Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:33:56.882Z] ] ***
[2026-09-29T10:34:11.109Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left).
[2026-09-29T10:34:11.109Z] ok: [an-a01n01]
[2026-09-29T10:34:11.109Z] 
[2026-09-29T10:34:11.109Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:34:11.109Z] ] ***
[2026-09-29T10:34:11.109Z] ok: [an-a01n01]
[2026-09-29T10:34:11.109Z] 
[2026-09-29T10:34:11.109Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:34:11.109Z] ] ***
[2026-09-29T10:34:11.109Z] ok: [an-a01n01]
[2026-09-29T10:34:11.109Z] 
[2026-09-29T10:34:11.109Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:34:11.109Z] ] ***
[2026-09-29T10:34:11.519Z] ok: [an-a01n01]
[2026-09-29T10:34:11.519Z] 
[2026-09-29T10:34:11.519Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:34:11.519Z] ] ***
[2026-09-29T10:34:12.390Z] ok: [an-a01n01]
[2026-09-29T10:34:12.390Z] 
[2026-09-29T10:34:12.390Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-29T10:34:12.390Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:34:12.390Z] 
[2026-09-29T10:34:12.390Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T10:37:23.675Z] ok: [an-a01n02 -> localhost]
[2026-09-29T10:37:23.675Z] ok: [an-a01n01 -> localhost]
[2026-09-29T10:37:23.675Z] 
[2026-09-29T10:37:23.675Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-29T10:37:23.675Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:37:23.675Z] 
[2026-09-29T10:37:23.675Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:37:23.675Z] ] ***
[2026-09-29T10:37:28.067Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:37:28.067Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:37:28.067Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:37:28.067Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:37:28.067Z] 
[2026-09-29T10:37:28.067Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:37:28.067Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:37:28.067Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:37:28.067Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:37:28.067Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:37:28.067Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:37:28.067Z] 
[2026-09-29T10:37:28.067Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:37:28.067Z] ] ***
[2026-09-29T10:37:28.939Z] ok: [an-a01n02]
[2026-09-29T10:37:28.939Z] ok: [an-a01n01]
[2026-09-29T10:37:28.939Z] 
[2026-09-29T10:37:28.939Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:37:28.939Z] ] ***
[2026-09-29T10:37:29.350Z] ok: [an-a01n02]
[2026-09-29T10:37:29.350Z] ok: [an-a01n01]
[2026-09-29T10:37:29.350Z] 
[2026-09-29T10:37:29.350Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:37:29.350Z] ] ***
[2026-09-29T10:37:30.222Z] ok: [an-a01n02]
[2026-09-29T10:37:30.222Z] ok: [an-a01n01]
[2026-09-29T10:37:30.222Z] 
[2026-09-29T10:37:30.222Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:37:30.222Z] ] ***
[2026-09-29T10:37:31.094Z] ok: [an-a01n02]
[2026-09-29T10:37:31.094Z] ok: [an-a01n01]
[2026-09-29T10:37:31.094Z] 
[2026-09-29T10:37:31.094Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:37:31.094Z] ] ***
[2026-09-29T10:37:31.505Z] ok: [an-a01n02]
[2026-09-29T10:37:31.505Z] ok: [an-a01n01]
[2026-09-29T10:37:31.505Z] 
[2026-09-29T10:37:31.505Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ******************
[2026-09-29T10:37:31.505Z] ok: [an-a01n01]
[2026-09-29T10:37:31.505Z] ok: [an-a01n02]
[2026-09-29T10:37:31.505Z] 
[2026-09-29T10:37:31.505Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:37:31.505Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:37:31.505Z] 
[2026-09-29T10:37:31.505Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:37:31.505Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:37:31.505Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:37:31.505Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:37:31.505Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:37:31.505Z]   echo "pcs:"
[2026-09-29T10:37:31.505Z]   echo "$pcsstatus"
[2026-09-29T10:37:31.505Z]   exit 1
[2026-09-29T10:37:31.505Z] fi
[2026-09-29T10:37:31.505Z] exit 0
[2026-09-29T10:37:31.505Z] ] ***
[2026-09-29T10:37:34.169Z] ok: [an-a01n02]
[2026-09-29T10:37:34.169Z] ok: [an-a01n01]
[2026-09-29T10:37:34.169Z] 
[2026-09-29T10:37:34.169Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:37:34.169Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:37:34.169Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:37:34.169Z] ret=0
[2026-09-29T10:37:34.169Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:37:34.169Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:37:34.169Z]   echo "$pcsstatus"
[2026-09-29T10:37:34.169Z]   ret=1
[2026-09-29T10:37:34.169Z] fi
[2026-09-29T10:37:34.169Z] echo "$hypervisor"
[2026-09-29T10:37:34.169Z] exit $ret
[2026-09-29T10:37:34.169Z] ] ***
[2026-09-29T10:37:36.155Z] ok: [an-a01n02]
[2026-09-29T10:37:36.155Z] ok: [an-a01n01]
[2026-09-29T10:37:36.155Z] 
[2026-09-29T10:37:36.155Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] *********************
[2026-09-29T10:37:36.155Z] ok: [an-a01n01]
[2026-09-29T10:37:36.155Z] ok: [an-a01n02]
[2026-09-29T10:37:36.155Z] 
[2026-09-29T10:37:36.155Z] TASK [will kill node with servers _raw_params=./kill_node.yml] *****************
[2026-09-29T10:37:36.155Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:37:36.155Z] 
[2026-09-29T10:37:36.155Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger
[2026-09-29T10:37:36.155Z] ] **********
[2026-09-29T10:37:36.566Z] skipping: [an-a01n02]
[2026-09-29T10:37:36.566Z] changed: [an-a01n01]
[2026-09-29T10:37:36.566Z] 
[2026-09-29T10:37:36.566Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] ***
[2026-09-29T10:37:36.977Z] skipping: [an-a01n01]
[2026-09-29T10:37:36.977Z] skipping: [an-a01n02]
[2026-09-29T10:37:36.977Z] 
[2026-09-29T10:37:36.977Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] **************
[2026-09-29T10:37:36.977Z] skipping: [an-a01n01]
[2026-09-29T10:37:36.977Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02
[2026-09-29T10:37:36.977Z] 
[2026-09-29T10:37:36.977Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}"
[2026-09-29T10:37:36.977Z] ] ***
[2026-09-29T10:38:48.199Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left).
[2026-09-29T10:38:48.199Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left).
[2026-09-29T10:38:48.199Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left).
[2026-09-29T10:38:48.199Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left).
[2026-09-29T10:38:48.199Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left).
[2026-09-29T10:38:48.199Z] ok: [an-a01n02]
[2026-09-29T10:38:48.199Z] 
[2026-09-29T10:38:48.200Z] TASK [check servers status _raw_params=./server_started.yml] *******************
[2026-09-29T10:38:48.200Z] skipping: [an-a01n01]
[2026-09-29T10:38:48.200Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02
[2026-09-29T10:38:48.200Z] 
[2026-09-29T10:38:48.200Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:38:48.200Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:38:48.200Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:38:48.200Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:38:48.200Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:38:48.200Z]   echo "pcs:"
[2026-09-29T10:38:48.200Z]   echo "$pcsstatus"
[2026-09-29T10:38:48.200Z]   exit 1
[2026-09-29T10:38:48.200Z] fi
[2026-09-29T10:38:48.200Z] exit 0
[2026-09-29T10:38:48.200Z] ] ***
[2026-09-29T10:39:47.617Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left).
[2026-09-29T10:39:47.617Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left).
[2026-09-29T10:39:47.617Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left).
[2026-09-29T10:39:47.617Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left).
[2026-09-29T10:39:47.617Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left).
[2026-09-29T10:39:47.617Z] ok: [an-a01n02]
[2026-09-29T10:39:47.617Z] 
[2026-09-29T10:39:47.617Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:39:47.617Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:39:47.617Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:39:47.617Z] ret=0
[2026-09-29T10:39:47.617Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:39:47.617Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:39:47.617Z]   echo "$pcsstatus"
[2026-09-29T10:39:47.617Z]   ret=1
[2026-09-29T10:39:47.617Z] fi
[2026-09-29T10:39:47.617Z] echo "$hypervisor"
[2026-09-29T10:39:47.617Z] exit $ret
[2026-09-29T10:39:47.617Z] ] ***
[2026-09-29T10:39:47.617Z] ok: [an-a01n02]
[2026-09-29T10:39:47.617Z] 
[2026-09-29T10:39:47.617Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ********
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01] => (item=1) 
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01] => (item=2) 
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01] => (item=3) 
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01] => (item=4) 
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01] => (item=5) 
[2026-09-29T10:39:47.617Z] skipping: [an-a01n01]
[2026-09-29T10:39:47.617Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1)
[2026-09-29T10:39:47.617Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2)
[2026-09-29T10:39:47.617Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3)
[2026-09-29T10:39:47.617Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4)
[2026-09-29T10:39:47.617Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5)
[2026-09-29T10:39:47.617Z] 
[2026-09-29T10:39:47.617Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:39:47.617Z] ] ***
[2026-09-29T10:39:48.225Z] ok: [an-a01n02]
[2026-09-29T10:39:48.225Z] 
[2026-09-29T10:39:48.225Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:39:48.225Z] ] ***
[2026-09-29T10:40:05.234Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left).
[2026-09-29T10:40:05.234Z] ok: [an-a01n02]
[2026-09-29T10:40:05.234Z] 
[2026-09-29T10:40:05.234Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:40:05.234Z] ] ***
[2026-09-29T10:40:05.234Z] ok: [an-a01n02]
[2026-09-29T10:40:05.234Z] 
[2026-09-29T10:40:05.234Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:40:05.234Z] ] ***
[2026-09-29T10:40:05.234Z] ok: [an-a01n02]
[2026-09-29T10:40:05.234Z] 
[2026-09-29T10:40:05.234Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T10:40:05.234Z] ] ***
[2026-09-29T10:40:05.234Z] ok: [an-a01n02]
[2026-09-29T10:40:05.234Z] 
[2026-09-29T10:40:05.234Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] *************
[2026-09-29T10:40:05.645Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02
[2026-09-29T10:40:05.645Z] 
[2026-09-29T10:40:05.645Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T10:43:16.981Z] ok: [an-a01n02 -> localhost]
[2026-09-29T10:43:16.981Z] ok: [an-a01n01 -> localhost]
[2026-09-29T10:43:16.981Z] 
[2026-09-29T10:43:16.981Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] ***********
[2026-09-29T10:43:16.981Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:16.981Z] 
[2026-09-29T10:43:16.981Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:43:16.981Z] ] ***
[2026-09-29T10:43:20.422Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:43:20.423Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:43:20.423Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:43:20.423Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:43:20.423Z] 
[2026-09-29T10:43:20.423Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:43:20.423Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T10:43:20.423Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:43:20.423Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:43:20.423Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:43:20.423Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:43:20.423Z] 
[2026-09-29T10:43:20.423Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:43:20.423Z] ] ***
[2026-09-29T10:43:21.293Z] ok: [an-a01n01]
[2026-09-29T10:43:21.293Z] ok: [an-a01n02]
[2026-09-29T10:43:21.293Z] 
[2026-09-29T10:43:21.293Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:43:21.293Z] ] ***
[2026-09-29T10:43:22.164Z] ok: [an-a01n01]
[2026-09-29T10:43:22.164Z] ok: [an-a01n02]
[2026-09-29T10:43:22.164Z] 
[2026-09-29T10:43:22.164Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:43:22.164Z] ] ***
[2026-09-29T10:43:22.575Z] ok: [an-a01n02]
[2026-09-29T10:43:22.575Z] ok: [an-a01n01]
[2026-09-29T10:43:22.575Z] 
[2026-09-29T10:43:22.575Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:43:22.575Z] ] ***
[2026-09-29T10:43:23.445Z] ok: [an-a01n01]
[2026-09-29T10:43:23.445Z] ok: [an-a01n02]
[2026-09-29T10:43:23.445Z] 
[2026-09-29T10:43:23.445Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:43:23.445Z] ] ***
[2026-09-29T10:43:24.315Z] ok: [an-a01n02]
[2026-09-29T10:43:24.315Z] ok: [an-a01n01]
[2026-09-29T10:43:24.315Z] 
[2026-09-29T10:43:24.315Z] PLAY [basic server tests] ******************************************************
[2026-09-29T10:43:24.315Z] 
[2026-09-29T10:43:24.315Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T10:43:26.299Z] ok: [an-a01n01]
[2026-09-29T10:43:26.299Z] ok: [an-a01n02]
[2026-09-29T10:43:26.299Z] 
[2026-09-29T10:43:26.299Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T10:43:26.299Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:26.299Z] 
[2026-09-29T10:43:26.299Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:43:26.299Z] ] ***
[2026-09-29T10:43:30.649Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:43:30.649Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:43:30.649Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:43:30.649Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:43:30.649Z] 
[2026-09-29T10:43:30.649Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-29T10:43:30.649Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:30.649Z] 
[2026-09-29T10:43:30.649Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-29T10:43:30.649Z] if [ -z "$storage" ]; then
[2026-09-29T10:43:30.649Z]   echo "Unable to detect shared storage"
[2026-09-29T10:43:30.649Z]   exit 1
[2026-09-29T10:43:30.649Z] fi
[2026-09-29T10:43:30.649Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T10:43:30.649Z] ] ***
[2026-09-29T10:43:37.308Z] changed: [an-a01n01]
[2026-09-29T10:43:37.308Z] changed: [an-a01n02]
[2026-09-29T10:43:37.308Z] 
[2026-09-29T10:43:37.308Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-29T10:43:37.308Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:37.308Z] 
[2026-09-29T10:43:37.308Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-29T10:43:37.308Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-29T10:43:37.308Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-29T10:43:37.308Z]   echo "Unable to detect all deploy iso"
[2026-09-29T10:43:37.308Z]   echo "Detected isos"
[2026-09-29T10:43:37.308Z]   echo "$isos"
[2026-09-29T10:43:37.308Z]   exit 1
[2026-09-29T10:43:37.308Z] fi
[2026-09-29T10:43:37.308Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-29T10:43:37.308Z] if [ -z "$installiso" ]; then
[2026-09-29T10:43:37.308Z]   echo "Unable to detect install iso"
[2026-09-29T10:43:37.308Z]   echo "Detected isos"
[2026-09-29T10:43:37.308Z]   echo "$isos"
[2026-09-29T10:43:37.308Z]   exit 1
[2026-09-29T10:43:37.308Z] fi
[2026-09-29T10:43:37.308Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T10:43:37.308Z] ] ***
[2026-09-29T10:43:43.949Z] changed: [an-a01n01]
[2026-09-29T10:43:43.949Z] changed: [an-a01n02]
[2026-09-29T10:43:43.949Z] 
[2026-09-29T10:43:43.949Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-29T10:43:43.949Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:43.949Z] 
[2026-09-29T10:43:43.949Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:43:43.949Z] ] ***
[2026-09-29T10:43:44.821Z] skipping: [an-a01n02]
[2026-09-29T10:43:44.821Z] ok: [an-a01n01]
[2026-09-29T10:43:44.821Z] 
[2026-09-29T10:43:44.821Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T10:43:44.821Z]   --ci-test \
[2026-09-29T10:43:44.821Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T10:43:44.821Z]   --os almalinux9 \
[2026-09-29T10:43:44.821Z]   --cpu 2 --ram 4G \
[2026-09-29T10:43:44.821Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T10:43:44.821Z]   --install-media {{ installisouuid }} \
[2026-09-29T10:43:44.821Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T10:43:44.821Z] ] ***
[2026-09-29T10:43:44.821Z] skipping: [an-a01n01]
[2026-09-29T10:43:44.821Z] skipping: [an-a01n02]
[2026-09-29T10:43:44.821Z] 
[2026-09-29T10:43:44.821Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T10:43:44.821Z] skipping: [an-a01n01]
[2026-09-29T10:43:44.821Z] skipping: [an-a01n02]
[2026-09-29T10:43:44.821Z] 
[2026-09-29T10:43:44.821Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:43:44.821Z] skipping: [an-a01n01]
[2026-09-29T10:43:44.821Z] skipping: [an-a01n02]
[2026-09-29T10:43:44.821Z] 
[2026-09-29T10:43:44.821Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T10:43:45.231Z] skipping: [an-a01n01]
[2026-09-29T10:43:45.231Z] skipping: [an-a01n02]
[2026-09-29T10:43:45.231Z] 
[2026-09-29T10:43:45.231Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T10:43:45.231Z] skipping: [an-a01n01]
[2026-09-29T10:43:45.231Z] skipping: [an-a01n02]
[2026-09-29T10:43:45.231Z] 
[2026-09-29T10:43:45.231Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] ***
[2026-09-29T10:43:45.231Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:45.231Z] 
[2026-09-29T10:43:45.231Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:43:45.231Z] ] ***
[2026-09-29T10:43:46.623Z] skipping: [an-a01n01]
[2026-09-29T10:43:46.623Z] ok: [an-a01n02]
[2026-09-29T10:43:46.623Z] 
[2026-09-29T10:43:46.623Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T10:43:46.623Z]   --ci-test \
[2026-09-29T10:43:46.623Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T10:43:46.623Z]   --os almalinux9 \
[2026-09-29T10:43:46.623Z]   --cpu 2 --ram 4G \
[2026-09-29T10:43:46.623Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T10:43:46.623Z]   --install-media {{ installisouuid }} \
[2026-09-29T10:43:46.623Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T10:43:46.623Z] ] ***
[2026-09-29T10:43:47.034Z] skipping: [an-a01n01]
[2026-09-29T10:43:47.034Z] skipping: [an-a01n02]
[2026-09-29T10:43:47.034Z] 
[2026-09-29T10:43:47.034Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T10:43:47.034Z] skipping: [an-a01n01]
[2026-09-29T10:43:47.034Z] skipping: [an-a01n02]
[2026-09-29T10:43:47.034Z] 
[2026-09-29T10:43:47.034Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:43:47.034Z] skipping: [an-a01n01]
[2026-09-29T10:43:47.034Z] skipping: [an-a01n02]
[2026-09-29T10:43:47.034Z] 
[2026-09-29T10:43:47.034Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T10:43:47.034Z] skipping: [an-a01n01]
[2026-09-29T10:43:47.034Z] skipping: [an-a01n02]
[2026-09-29T10:43:47.034Z] 
[2026-09-29T10:43:47.034Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T10:43:47.034Z] skipping: [an-a01n01]
[2026-09-29T10:43:47.034Z] skipping: [an-a01n02]
[2026-09-29T10:43:47.034Z] 
[2026-09-29T10:43:47.034Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-29T10:43:47.445Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:47.445Z] 
[2026-09-29T10:43:47.445Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:43:47.445Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:43:47.445Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:43:47.445Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:43:47.445Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:43:47.445Z]   echo "pcs:"
[2026-09-29T10:43:47.445Z]   echo "$pcsstatus"
[2026-09-29T10:43:47.445Z]   exit 1
[2026-09-29T10:43:47.445Z] fi
[2026-09-29T10:43:47.445Z] exit 0
[2026-09-29T10:43:47.445Z] ] ***
[2026-09-29T10:43:49.428Z] ok: [an-a01n02]
[2026-09-29T10:43:49.428Z] ok: [an-a01n01]
[2026-09-29T10:43:49.428Z] 
[2026-09-29T10:43:49.428Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:43:49.428Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:43:49.428Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:43:49.428Z] ret=0
[2026-09-29T10:43:49.428Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:43:49.428Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:43:49.428Z]   echo "$pcsstatus"
[2026-09-29T10:43:49.428Z]   ret=1
[2026-09-29T10:43:49.428Z] fi
[2026-09-29T10:43:49.428Z] echo "$hypervisor"
[2026-09-29T10:43:49.428Z] exit $ret
[2026-09-29T10:43:49.428Z] ] ***
[2026-09-29T10:43:51.607Z] ok: [an-a01n02]
[2026-09-29T10:43:51.607Z] ok: [an-a01n01]
[2026-09-29T10:43:51.607Z] 
[2026-09-29T10:43:51.607Z] PLAY [delete extra server tests] ***********************************************
[2026-09-29T10:43:51.607Z] 
[2026-09-29T10:43:51.607Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T10:43:53.591Z] ok: [an-a01n01]
[2026-09-29T10:43:53.591Z] ok: [an-a01n02]
[2026-09-29T10:43:53.591Z] 
[2026-09-29T10:43:53.591Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T10:43:54.001Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:43:54.001Z] 
[2026-09-29T10:43:54.001Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:43:54.001Z] ] ***
[2026-09-29T10:43:58.355Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:43:58.355Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:43:58.355Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:43:58.355Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:43:58.355Z] 
[2026-09-29T10:43:58.355Z] TASK [delete servers _raw_params=../../common/delete_server.yml] ***************
[2026-09-29T10:43:58.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T10:43:58.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T10:43:58.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T10:43:58.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T10:43:58.355Z] 
[2026-09-29T10:43:58.355Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:43:58.355Z] ] ***
[2026-09-29T10:44:00.339Z] skipping: [an-a01n01]
[2026-09-29T10:44:00.339Z] ok: [an-a01n02]
[2026-09-29T10:44:00.339Z] 
[2026-09-29T10:44:00.339Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T10:44:00.339Z]   --force \
[2026-09-29T10:44:00.339Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T10:44:00.339Z] ] ***
[2026-09-29T10:44:10.125Z] skipping: [an-a01n01]
[2026-09-29T10:44:10.125Z] changed: [an-a01n02]
[2026-09-29T10:44:10.125Z] 
[2026-09-29T10:44:10.125Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:44:10.125Z] ] ***
[2026-09-29T10:44:34.686Z] skipping: [an-a01n01]
[2026-09-29T10:44:34.686Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left).
[2026-09-29T10:44:34.686Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left).
[2026-09-29T10:44:34.686Z] ok: [an-a01n02]
[2026-09-29T10:44:34.686Z] 
[2026-09-29T10:44:34.686Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:44:34.686Z] ] ***
[2026-09-29T10:44:34.686Z] skipping: [an-a01n01]
[2026-09-29T10:44:34.686Z] ok: [an-a01n02]
[2026-09-29T10:44:34.686Z] 
[2026-09-29T10:44:34.686Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T10:44:34.686Z] ] ***
[2026-09-29T10:44:35.556Z] skipping: [an-a01n01]
[2026-09-29T10:44:35.557Z] ok: [an-a01n02]
[2026-09-29T10:44:35.557Z] 
[2026-09-29T10:44:35.557Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:44:35.557Z] ] ***
[2026-09-29T10:44:36.950Z] skipping: [an-a01n01]
[2026-09-29T10:44:36.950Z] ok: [an-a01n02]
[2026-09-29T10:44:36.950Z] 
[2026-09-29T10:44:36.950Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T10:44:36.950Z]   --force \
[2026-09-29T10:44:36.950Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T10:44:36.950Z] ] ***
[2026-09-29T10:44:45.034Z] skipping: [an-a01n01]
[2026-09-29T10:44:45.034Z] changed: [an-a01n02]
[2026-09-29T10:44:45.034Z] 
[2026-09-29T10:44:45.034Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:44:45.034Z] ] ***
[2026-09-29T10:45:56.061Z] skipping: [an-a01n01]
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left).
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left).
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left).
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left).
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left).
[2026-09-29T10:45:56.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left).
[2026-09-29T10:45:56.061Z] ok: [an-a01n02]
[2026-09-29T10:45:56.061Z] 
[2026-09-29T10:45:56.061Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:45:56.061Z] ] ***
[2026-09-29T10:45:56.472Z] skipping: [an-a01n01]
[2026-09-29T10:45:56.472Z] ok: [an-a01n02]
[2026-09-29T10:45:56.472Z] 
[2026-09-29T10:45:56.472Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T10:45:56.472Z] ] ***
[2026-09-29T10:45:57.344Z] skipping: [an-a01n01]
[2026-09-29T10:45:57.344Z] ok: [an-a01n02]
[2026-09-29T10:45:57.344Z] 
[2026-09-29T10:45:57.344Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:45:57.344Z] ] ***
[2026-09-29T10:45:59.329Z] skipping: [an-a01n01]
[2026-09-29T10:45:59.329Z] ok: [an-a01n02]
[2026-09-29T10:45:59.329Z] 
[2026-09-29T10:45:59.329Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T10:45:59.329Z]   --force \
[2026-09-29T10:45:59.329Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T10:45:59.329Z] ] ***
[2026-09-29T10:46:07.415Z] skipping: [an-a01n01]
[2026-09-29T10:46:07.415Z] changed: [an-a01n02]
[2026-09-29T10:46:07.415Z] 
[2026-09-29T10:46:07.415Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:46:07.415Z] ] ***
[2026-09-29T10:47:07.027Z] skipping: [an-a01n01]
[2026-09-29T10:47:07.027Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left).
[2026-09-29T10:47:07.027Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left).
[2026-09-29T10:47:07.027Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left).
[2026-09-29T10:47:07.027Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left).
[2026-09-29T10:47:07.027Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left).
[2026-09-29T10:47:07.027Z] ok: [an-a01n02]
[2026-09-29T10:47:07.027Z] 
[2026-09-29T10:47:07.027Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:47:07.027Z] ] ***
[2026-09-29T10:47:07.437Z] skipping: [an-a01n01]
[2026-09-29T10:47:07.437Z] ok: [an-a01n02]
[2026-09-29T10:47:07.437Z] 
[2026-09-29T10:47:07.437Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T10:47:07.437Z] ] ***
[2026-09-29T10:47:08.308Z] skipping: [an-a01n01]
[2026-09-29T10:47:08.308Z] ok: [an-a01n02]
[2026-09-29T10:47:08.308Z] 
[2026-09-29T10:47:08.308Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:47:08.308Z] ] ***
[2026-09-29T10:47:10.291Z] skipping: [an-a01n01]
[2026-09-29T10:47:10.291Z] ok: [an-a01n02]
[2026-09-29T10:47:10.291Z] 
[2026-09-29T10:47:10.291Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T10:47:10.291Z]   --force \
[2026-09-29T10:47:10.291Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T10:47:10.291Z] ] ***
[2026-09-29T10:47:18.374Z] skipping: [an-a01n01]
[2026-09-29T10:47:18.374Z] changed: [an-a01n02]
[2026-09-29T10:47:18.374Z] 
[2026-09-29T10:47:18.374Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:47:18.374Z] ] ***
[2026-09-29T10:48:29.607Z] skipping: [an-a01n01]
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left).
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left).
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left).
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left).
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left).
[2026-09-29T10:48:29.607Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (55 retries left).
[2026-09-29T10:48:29.607Z] ok: [an-a01n02]
[2026-09-29T10:48:29.607Z] 
[2026-09-29T10:48:29.607Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:48:29.607Z] ] ***
[2026-09-29T10:48:30.018Z] skipping: [an-a01n01]
[2026-09-29T10:48:30.018Z] ok: [an-a01n02]
[2026-09-29T10:48:30.018Z] 
[2026-09-29T10:48:30.018Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T10:48:30.018Z] ] ***
[2026-09-29T10:48:30.429Z] skipping: [an-a01n01]
[2026-09-29T10:48:30.429Z] ok: [an-a01n02]
[2026-09-29T10:48:30.429Z] 
[2026-09-29T10:48:30.429Z] PLAY [create server for DR testing] ********************************************
[2026-09-29T10:48:30.429Z] 
[2026-09-29T10:48:30.429Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T10:48:33.097Z] ok: [an-a01n02]
[2026-09-29T10:48:33.097Z] ok: [an-a01n01]
[2026-09-29T10:48:33.097Z] 
[2026-09-29T10:48:33.097Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T10:48:33.097Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T10:48:33.097Z] 
[2026-09-29T10:48:33.097Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T10:48:33.097Z] ] ***
[2026-09-29T10:48:37.452Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T10:48:37.452Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T10:48:37.452Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T10:48:37.452Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T10:48:37.452Z] 
[2026-09-29T10:48:37.452Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-29T10:48:37.452Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-29T10:48:37.452Z] 
[2026-09-29T10:48:37.452Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-29T10:48:37.452Z] if [ -z "$storage" ]; then
[2026-09-29T10:48:37.452Z]   echo "Unable to detect shared storage"
[2026-09-29T10:48:37.452Z]   exit 1
[2026-09-29T10:48:37.452Z] fi
[2026-09-29T10:48:37.452Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T10:48:37.452Z] ] ***
[2026-09-29T10:48:43.020Z] changed: [an-a01n01]
[2026-09-29T10:48:43.020Z] changed: [an-a01n02]
[2026-09-29T10:48:43.020Z] 
[2026-09-29T10:48:43.020Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-29T10:48:43.020Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-29T10:48:43.020Z] 
[2026-09-29T10:48:43.020Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-29T10:48:43.020Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-29T10:48:43.020Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-29T10:48:43.020Z]   echo "Unable to detect all deploy iso"
[2026-09-29T10:48:43.020Z]   echo "Detected isos"
[2026-09-29T10:48:43.020Z]   echo "$isos"
[2026-09-29T10:48:43.020Z]   exit 1
[2026-09-29T10:48:43.020Z] fi
[2026-09-29T10:48:43.020Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-29T10:48:43.020Z] if [ -z "$installiso" ]; then
[2026-09-29T10:48:43.020Z]   echo "Unable to detect install iso"
[2026-09-29T10:48:43.020Z]   echo "Detected isos"
[2026-09-29T10:48:43.020Z]   echo "$isos"
[2026-09-29T10:48:43.020Z]   exit 1
[2026-09-29T10:48:43.020Z] fi
[2026-09-29T10:48:43.020Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T10:48:43.020Z] ] ***
[2026-09-29T10:48:48.426Z] changed: [an-a01n02]
[2026-09-29T10:48:48.426Z] changed: [an-a01n01]
[2026-09-29T10:48:48.426Z] 
[2026-09-29T10:48:48.426Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-29T10:48:48.426Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-29T10:48:48.426Z] 
[2026-09-29T10:48:48.426Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T10:48:48.426Z] ] ***
[2026-09-29T10:48:50.411Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.411Z] ok: [an-a01n01]
[2026-09-29T10:48:50.411Z] 
[2026-09-29T10:48:50.411Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T10:48:50.411Z]   --ci-test \
[2026-09-29T10:48:50.411Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T10:48:50.411Z]   --os almalinux9 \
[2026-09-29T10:48:50.411Z]   --cpu 2 --ram 4G \
[2026-09-29T10:48:50.411Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T10:48:50.411Z]   --install-media {{ installisouuid }} \
[2026-09-29T10:48:50.411Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T10:48:50.411Z] ] ***
[2026-09-29T10:48:50.411Z] skipping: [an-a01n01]
[2026-09-29T10:48:50.411Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.411Z] 
[2026-09-29T10:48:50.411Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T10:48:50.411Z] skipping: [an-a01n01]
[2026-09-29T10:48:50.411Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.411Z] 
[2026-09-29T10:48:50.411Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T10:48:50.411Z] skipping: [an-a01n01]
[2026-09-29T10:48:50.411Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.411Z] 
[2026-09-29T10:48:50.411Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T10:48:50.821Z] skipping: [an-a01n01]
[2026-09-29T10:48:50.821Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.821Z] 
[2026-09-29T10:48:50.821Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T10:48:50.821Z] skipping: [an-a01n01]
[2026-09-29T10:48:50.821Z] skipping: [an-a01n02]
[2026-09-29T10:48:50.821Z] 
[2026-09-29T10:48:50.821Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-29T10:48:50.821Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-29T10:48:50.821Z] 
[2026-09-29T10:48:50.821Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:48:50.821Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T10:48:50.821Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T10:48:50.821Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T10:48:50.821Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T10:48:50.821Z]   echo "pcs:"
[2026-09-29T10:48:50.821Z]   echo "$pcsstatus"
[2026-09-29T10:48:50.821Z]   exit 1
[2026-09-29T10:48:50.821Z] fi
[2026-09-29T10:48:50.821Z] exit 0
[2026-09-29T10:48:50.821Z] ] ***
[2026-09-29T10:48:52.805Z] ok: [an-a01n01]
[2026-09-29T10:48:52.805Z] ok: [an-a01n02]
[2026-09-29T10:48:52.805Z] 
[2026-09-29T10:48:52.805Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T10:48:52.805Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T10:48:52.805Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T10:48:52.805Z] ret=0
[2026-09-29T10:48:52.805Z] if [ "$servers"  != "1" ]; then
[2026-09-29T10:48:52.805Z]   echo "Not all servers are running on the same node!"
[2026-09-29T10:48:52.805Z]   echo "$pcsstatus"
[2026-09-29T10:48:52.805Z]   ret=1
[2026-09-29T10:48:52.805Z] fi
[2026-09-29T10:48:52.805Z] echo "$hypervisor"
[2026-09-29T10:48:52.805Z] exit $ret
[2026-09-29T10:48:52.805Z] ] ***
[2026-09-29T10:48:54.791Z] ok: [an-a01n02]
[2026-09-29T10:48:54.791Z] ok: [an-a01n01]
[2026-09-29T10:48:54.791Z] 
[2026-09-29T10:48:54.791Z] PLAY [DR tests] ****************************************************************
[2026-09-29T10:48:54.791Z] 
[2026-09-29T10:48:54.791Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T10:48:57.646Z] ok: [an-a01dr01]
[2026-09-29T10:48:57.646Z] 
[2026-09-29T10:48:57.646Z] TASK [check DR is correctly deployed _raw_params=anvil-manage-host -vv --log-secure --check-configured
[2026-09-29T10:48:57.646Z] ] ***
[2026-09-29T10:49:01.091Z] changed: [an-a01dr01]
[2026-09-29T10:49:01.091Z] 
[2026-09-29T10:49:01.091Z] TASK [check DR host is registered in the DB _raw_params=drhostname=$(anvil-manage-dr -vv --log-secure --show | grep -A1 "DR Hosts" | grep Name | sed -e 's#.*\[##g' -e 's#\].*##g')
[2026-09-29T10:49:01.091Z] echo "DB registered hostname: $drhostname"
[2026-09-29T10:49:01.091Z] echo "current hostname: $(hostname)"
[2026-09-29T10:49:01.091Z] if [ "$drhostname" != "$(hostname)" ]; then
[2026-09-29T10:49:01.091Z]   echo "DR hostnames are not matching"
[2026-09-29T10:49:01.091Z]   exit 1
[2026-09-29T10:49:01.091Z] fi
[2026-09-29T10:49:01.091Z] if [ "$drhostname" != "an-a01dr01.ci.alteeve.com" ]; then
[2026-09-29T10:49:01.091Z]   echo "DR hostnames are not matching drnodes an-a01dr01"
[2026-09-29T10:49:01.091Z]   exit 1
[2026-09-29T10:49:01.091Z] fi
[2026-09-29T10:49:01.091Z] ] ***
[2026-09-29T10:49:05.442Z] changed: [an-a01dr01]
[2026-09-29T10:49:05.442Z] 
[2026-09-29T10:49:05.442Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr -vv --log-secure --show  | grep -q "No linked DR hosts"; then
[2026-09-29T10:49:05.442Z]   echo "Detected a linked DR host!"
[2026-09-29T10:49:05.442Z]   exit 1
[2026-09-29T10:49:05.442Z] fi
[2026-09-29T10:49:05.442Z] ] ***
[2026-09-29T10:49:10.848Z] changed: [an-a01dr01]
[2026-09-29T10:49:10.848Z] 
[2026-09-29T10:49:10.848Z] TASK [link DR host to anvil _raw_params=anvil-manage-dr -vv --log-secure --dr-host $(hostname) --link --anvil an-anvil-01
[2026-09-29T10:49:10.848Z] ] ***
[2026-09-29T10:49:15.199Z] changed: [an-a01dr01]
[2026-09-29T10:49:15.199Z] 
[2026-09-29T10:49:15.199Z] TASK [verify DR host is linked to anvil _raw_params=if ! anvil-manage-dr -vv --log-secure --show | grep -q "Linked"; then
[2026-09-29T10:49:15.199Z]   echo "DR host failed to link to anvil"
[2026-09-29T10:49:15.199Z]   exit 1
[2026-09-29T10:49:15.199Z] fi
[2026-09-29T10:49:15.199Z] ] ***
[2026-09-29T10:49:19.548Z] changed: [an-a01dr01]
[2026-09-29T10:49:19.548Z] 
[2026-09-29T10:49:19.548Z] TASK [link DR vg to storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##')
[2026-09-29T10:49:19.548Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}')
[2026-09-29T10:49:19.548Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then
[2026-09-29T10:49:19.548Z]   echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]"
[2026-09-29T10:49:19.548Z]   exit 1
[2026-09-29T10:49:19.548Z] fi
[2026-09-29T10:49:19.548Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --add --member "$localvguuid"; then
[2026-09-29T10:49:19.548Z]   echo "Unable to link local vg uuid to storage group"
[2026-09-29T10:49:19.548Z]   exit 1
[2026-09-29T10:49:19.548Z] fi
[2026-09-29T10:49:19.548Z] ] ***
[2026-09-29T10:49:26.185Z] changed: [an-a01dr01]
[2026-09-29T10:49:26.185Z] 
[2026-09-29T10:49:26.185Z] TASK [verify DR vg is linked to storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 3 ]; then
[2026-09-29T10:49:26.185Z]   echo "Local vg does not appear to be linked to storage group"
[2026-09-29T10:49:26.185Z]   exit 1
[2026-09-29T10:49:26.185Z] fi
[2026-09-29T10:49:26.185Z] ] ***
[2026-09-29T10:49:29.627Z] changed: [an-a01dr01]
[2026-09-29T10:49:29.627Z] 
[2026-09-29T10:49:29.627Z] TASK [protect an-test-deploy1 short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol short-throw --dr-host $(hostname) --Yes; then
[2026-09-29T10:49:29.627Z]   echo "Unable to add protection to server an-test-deploy1"
[2026-09-29T10:49:29.627Z]   exit 1
[2026-09-29T10:49:29.627Z] fi
[2026-09-29T10:49:29.627Z] ] ***
[2026-09-29T10:49:35.032Z] changed: [an-a01dr01]
[2026-09-29T10:49:35.032Z] 
[2026-09-29T10:49:35.032Z] TASK [verify an-test-deploy1 DR drbd status short-throw _raw_params=../../common/wait_server_drbd_dr_sync.yml] ***
[2026-09-29T10:49:35.032Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01
[2026-09-29T10:49:35.032Z] 
[2026-09-29T10:49:35.032Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-29T10:49:35.032Z]   exit 1
[2026-09-29T10:49:35.032Z] fi
[2026-09-29T10:49:35.032Z] ] ***
[2026-09-29T10:50:34.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left).
[2026-09-29T10:50:34.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left).
[2026-09-29T10:50:34.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left).
[2026-09-29T10:50:34.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (117 retries left).
[2026-09-29T10:50:34.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (116 retries left).
[2026-09-29T10:50:34.454Z] ok: [an-a01dr01]
[2026-09-29T10:50:34.454Z] 
[2026-09-29T10:50:34.454Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in
[2026-09-29T10:50:34.454Z]   short-throw)
[2026-09-29T10:50:34.454Z]     expected=2
[2026-09-29T10:50:34.454Z]     protocol=A
[2026-09-29T10:50:34.454Z]   ;;
[2026-09-29T10:50:34.454Z]   sync)
[2026-09-29T10:50:34.454Z]     expected=3
[2026-09-29T10:50:34.454Z]     protocol=C
[2026-09-29T10:50:34.454Z]   ;;
[2026-09-29T10:50:34.454Z]   *)
[2026-09-29T10:50:34.454Z]     echo "unknown DRBD protection method"
[2026-09-29T10:50:34.454Z]     exit 1
[2026-09-29T10:50:34.454Z]   ;;
[2026-09-29T10:50:34.454Z] esac
[2026-09-29T10:50:34.454Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then
[2026-09-29T10:50:34.454Z]   echo "Unable to confirm correct DRBD protocol"
[2026-09-29T10:50:34.454Z]   exit 1
[2026-09-29T10:50:34.454Z] fi
[2026-09-29T10:50:34.454Z] ] ***
[2026-09-29T10:50:34.454Z] changed: [an-a01dr01]
[2026-09-29T10:50:34.454Z] 
[2026-09-29T10:50:34.454Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:50:34.454Z] ] ***
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (107 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (106 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (105 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (104 retries left).
[2026-09-29T10:53:45.794Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (103 retries left).
[2026-09-29T10:53:45.795Z] ok: [an-a01dr01]
[2026-09-29T10:53:45.795Z] 
[2026-09-29T10:53:45.795Z] TASK [verify disconnect/update/connect features short-throw _raw_params=../../common/check_server_drbd_dr_management.yml] ***
[2026-09-29T10:53:45.795Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01
[2026-09-29T10:53:45.795Z] 
[2026-09-29T10:53:45.795Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:53:45.795Z] ] ***
[2026-09-29T10:53:45.795Z] ok: [an-a01dr01]
[2026-09-29T10:53:45.795Z] 
[2026-09-29T10:53:45.795Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then
[2026-09-29T10:53:45.795Z]   echo "Unable to disconnect server from DR replication"
[2026-09-29T10:53:45.795Z]   exit 1
[2026-09-29T10:53:45.795Z] fi
[2026-09-29T10:53:45.795Z] ] ***
[2026-09-29T10:53:48.457Z] changed: [an-a01dr01]
[2026-09-29T10:53:48.457Z] 
[2026-09-29T10:53:48.457Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:53:48.457Z] ] ***
[2026-09-29T10:54:12.817Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-29T10:54:12.817Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left).
[2026-09-29T10:54:12.817Z] ok: [an-a01dr01]
[2026-09-29T10:54:12.817Z] 
[2026-09-29T10:54:12.817Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock"
[2026-09-29T10:54:12.817Z] sleep 120
[2026-09-29T10:54:12.817Z] ] ***
[2026-09-29T10:56:23.389Z] changed: [an-a01dr01 -> localhost]
[2026-09-29T10:56:23.389Z] 
[2026-09-29T10:56:23.389Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then
[2026-09-29T10:56:23.389Z]   echo "Unable to update server DR replications"
[2026-09-29T10:56:23.389Z]   exit 1
[2026-09-29T10:56:23.389Z] fi
[2026-09-29T10:56:23.389Z] ] ***
[2026-09-29T10:56:23.389Z] changed: [an-a01dr01]
[2026-09-29T10:56:23.389Z] 
[2026-09-29T10:56:23.389Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:56:23.389Z] ] ***
[2026-09-29T10:56:58.199Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-29T10:56:58.199Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-29T10:56:58.199Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left).
[2026-09-29T10:56:58.199Z] ok: [an-a01dr01]
[2026-09-29T10:56:58.199Z] 
[2026-09-29T10:56:58.199Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:56:58.199Z] ] ***
[2026-09-29T10:57:18.718Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-29T10:57:18.718Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left).
[2026-09-29T10:57:18.718Z] ok: [an-a01dr01]
[2026-09-29T10:57:18.718Z] 
[2026-09-29T10:57:18.718Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then
[2026-09-29T10:57:18.718Z]   echo "Unable to (re)connect server from DR replication"
[2026-09-29T10:57:18.718Z]   exit 1
[2026-09-29T10:57:18.718Z] fi
[2026-09-29T10:57:18.718Z] ] ***
[2026-09-29T10:57:22.161Z] changed: [an-a01dr01]
[2026-09-29T10:57:22.161Z] 
[2026-09-29T10:57:22.161Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:57:22.161Z] ] ***
[2026-09-29T10:58:12.070Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-29T10:58:12.070Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-29T10:58:12.070Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left).
[2026-09-29T10:58:12.070Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (117 retries left).
[2026-09-29T10:58:12.070Z] ok: [an-a01dr01]
[2026-09-29T10:58:12.070Z] 
[2026-09-29T10:58:12.070Z] TASK [remove an-test-deploy1 protection short-throw _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then
[2026-09-29T10:58:12.070Z]   echo "Unable to remove protection from server an-test-deploy1"
[2026-09-29T10:58:12.070Z]   exit 1
[2026-09-29T10:58:12.070Z] fi
[2026-09-29T10:58:12.070Z] ] ***
[2026-09-29T10:58:12.070Z] changed: [an-a01dr01]
[2026-09-29T10:58:12.070Z] 
[2026-09-29T10:58:12.070Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw _raw_params=../../common/wait_server_drbd_dr_clean.yml] ***
[2026-09-29T10:58:12.070Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01
[2026-09-29T10:58:12.070Z] 
[2026-09-29T10:58:12.070Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T10:58:12.070Z] ] ***
[2026-09-29T10:58:46.876Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-29T10:58:46.877Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left).
[2026-09-29T10:58:46.877Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left).
[2026-09-29T10:58:46.877Z] ok: [an-a01dr01]
[2026-09-29T10:58:46.877Z] 
[2026-09-29T10:58:46.877Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T10:58:46.877Z] ] ***
[2026-09-29T10:58:46.877Z] ok: [an-a01dr01]
[2026-09-29T10:58:46.877Z] 
[2026-09-29T10:58:46.877Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-29T10:58:46.877Z]   exit 1
[2026-09-29T10:58:46.877Z] fi
[2026-09-29T10:58:46.877Z] ] ***
[2026-09-29T10:59:16.005Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left).
[2026-09-29T10:59:16.005Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left).
[2026-09-29T10:59:16.005Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left).
[2026-09-29T10:59:16.005Z] ok: [an-a01dr01]
[2026-09-29T10:59:16.005Z] 
[2026-09-29T10:59:16.005Z] TASK [check drbd sync status on nodes after disconnect short-throw _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then
[2026-09-29T10:59:16.005Z]   echo "drbdadm status still reports connection to DR node (#888)"
[2026-09-29T10:59:16.005Z]   #exit 1
[2026-09-29T10:59:16.005Z] fi
[2026-09-29T10:59:16.005Z] ] ***
[2026-09-29T10:59:16.874Z] changed: [an-a01dr01 -> an-a01n01]
[2026-09-29T10:59:16.874Z] 
[2026-09-29T10:59:16.874Z] TASK [protect an-test-deploy1 sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --protect --protocol sync --dr-host $(hostname) --Yes; then
[2026-09-29T10:59:16.874Z]   echo "Unable to add protection to server an-test-deploy1"
[2026-09-29T10:59:16.874Z]   exit 1
[2026-09-29T10:59:16.874Z] fi
[2026-09-29T10:59:16.874Z] ] ***
[2026-09-29T10:59:22.275Z] changed: [an-a01dr01]
[2026-09-29T10:59:22.275Z] 
[2026-09-29T10:59:22.275Z] TASK [verify an-test-deploy1 DR drbd status sync _raw_params=../../common/wait_server_drbd_dr_sync.yml] ***
[2026-09-29T10:59:22.275Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01
[2026-09-29T10:59:22.275Z] 
[2026-09-29T10:59:22.275Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created _raw_params=if [ ! -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-29T10:59:22.275Z]   exit 1
[2026-09-29T10:59:22.275Z] fi
[2026-09-29T10:59:22.275Z] ] ***
[2026-09-29T10:59:57.089Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left).
[2026-09-29T10:59:57.089Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left).
[2026-09-29T10:59:57.089Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left).
[2026-09-29T10:59:57.089Z] ok: [an-a01dr01]
[2026-09-29T10:59:57.089Z] 
[2026-09-29T10:59:57.089Z] TASK [Check correct drbd protocol is configured _raw_params=case {{ protection }} in
[2026-09-29T10:59:57.089Z]   short-throw)
[2026-09-29T10:59:57.089Z]     expected=2
[2026-09-29T10:59:57.089Z]     protocol=A
[2026-09-29T10:59:57.089Z]   ;;
[2026-09-29T10:59:57.089Z]   sync)
[2026-09-29T10:59:57.089Z]     expected=3
[2026-09-29T10:59:57.089Z]     protocol=C
[2026-09-29T10:59:57.089Z]   ;;
[2026-09-29T10:59:57.089Z]   *)
[2026-09-29T10:59:57.089Z]     echo "unknown DRBD protection method"
[2026-09-29T10:59:57.089Z]     exit 1
[2026-09-29T10:59:57.089Z]   ;;
[2026-09-29T10:59:57.089Z] esac
[2026-09-29T10:59:57.089Z] if [ $(cat /etc/drbd.d/an-test-deploy{{ servernum }}.res | grep "protocol $protocol;" | wc -l) != $expected ]; then
[2026-09-29T10:59:57.089Z]   echo "Unable to confirm correct DRBD protocol"
[2026-09-29T10:59:57.089Z]   exit 1
[2026-09-29T10:59:57.089Z] fi
[2026-09-29T10:59:57.089Z] ] ***
[2026-09-29T10:59:57.089Z] changed: [an-a01dr01]
[2026-09-29T10:59:57.089Z] 
[2026-09-29T10:59:57.089Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T10:59:57.089Z] ] ***
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (108 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (107 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (106 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (105 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (104 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (103 retries left).
[2026-09-29T11:03:23.501Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (102 retries left).
[2026-09-29T11:03:23.501Z] ok: [an-a01dr01]
[2026-09-29T11:03:23.501Z] 
[2026-09-29T11:03:23.501Z] TASK [verify disconnect/update/connect features sync _raw_params=../../common/check_server_drbd_dr_management.yml] ***
[2026-09-29T11:03:23.501Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01
[2026-09-29T11:03:23.501Z] 
[2026-09-29T11:03:23.501Z] TASK [Verify an-test-deploy1 drbd storage is synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T11:03:23.501Z] ] ***
[2026-09-29T11:03:23.501Z] ok: [an-a01dr01]
[2026-09-29T11:03:23.501Z] 
[2026-09-29T11:03:23.501Z] TASK [Disconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --disconnect --dr-host $(hostname) --Yes; then
[2026-09-29T11:03:23.501Z]   echo "Unable to disconnect server from DR replication"
[2026-09-29T11:03:23.501Z]   exit 1
[2026-09-29T11:03:23.501Z] fi
[2026-09-29T11:03:23.501Z] ] ***
[2026-09-29T11:03:23.912Z] changed: [an-a01dr01]
[2026-09-29T11:03:23.912Z] 
[2026-09-29T11:03:23.912Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:03:23.912Z] ] ***
[2026-09-29T11:03:48.431Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-29T11:03:48.431Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left).
[2026-09-29T11:03:48.431Z] ok: [an-a01dr01]
[2026-09-29T11:03:48.431Z] 
[2026-09-29T11:03:48.431Z] TASK [Inject random data to an-test-deploy1 _raw_params=ssh an-test-deploy{{ servernum }} "dd if=/dev/urandom of=/tmp/random_test_data bs=1G count=1 iflag=fullblock"
[2026-09-29T11:03:48.431Z] sleep 120
[2026-09-29T11:03:48.431Z] ] ***
[2026-09-29T11:05:58.999Z] changed: [an-a01dr01 -> localhost]
[2026-09-29T11:05:58.999Z] 
[2026-09-29T11:05:58.999Z] TASK [Update an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --update --dr-host $(hostname) --Yes; then
[2026-09-29T11:05:58.999Z]   echo "Unable to update server DR replications"
[2026-09-29T11:05:58.999Z]   exit 1
[2026-09-29T11:05:58.999Z] fi
[2026-09-29T11:05:58.999Z] ] ***
[2026-09-29T11:05:58.999Z] changed: [an-a01dr01]
[2026-09-29T11:05:58.999Z] 
[2026-09-29T11:05:58.999Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:05:58.999Z] ] ***
[2026-09-29T11:06:23.369Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-29T11:06:23.369Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-29T11:06:23.369Z] ok: [an-a01dr01]
[2026-09-29T11:06:23.369Z] 
[2026-09-29T11:06:23.369Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:06:23.369Z] ] ***
[2026-09-29T11:06:31.454Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left).
[2026-09-29T11:06:31.454Z] ok: [an-a01dr01]
[2026-09-29T11:06:31.454Z] 
[2026-09-29T11:06:31.454Z] TASK [Reconnect an-test-deploy1 drbd storage _raw_params=if ! anvil-manage-dr --server an-test-deploy{{ servernum }} --connect --dr-host $(hostname) --Yes; then
[2026-09-29T11:06:31.454Z]   echo "Unable to (re)connect server from DR replication"
[2026-09-29T11:06:31.454Z]   exit 1
[2026-09-29T11:06:31.454Z] fi
[2026-09-29T11:06:31.454Z] ] ***
[2026-09-29T11:06:36.860Z] changed: [an-a01dr01]
[2026-09-29T11:06:36.860Z] 
[2026-09-29T11:06:36.860Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:06:36.860Z] ] ***
[2026-09-29T11:07:26.734Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left).
[2026-09-29T11:07:26.735Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left).
[2026-09-29T11:07:26.735Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left).
[2026-09-29T11:07:26.735Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (117 retries left).
[2026-09-29T11:07:26.735Z] ok: [an-a01dr01]
[2026-09-29T11:07:26.735Z] 
[2026-09-29T11:07:26.735Z] TASK [remove an-test-deploy1 protection sync _raw_params=if ! anvil-manage-dr --server an-test-deploy1 --remove --dr-host $(hostname) --Yes; then
[2026-09-29T11:07:26.735Z]   echo "Unable to remove protection from server an-test-deploy1"
[2026-09-29T11:07:26.735Z]   exit 1
[2026-09-29T11:07:26.735Z] fi
[2026-09-29T11:07:26.735Z] ] ***
[2026-09-29T11:07:26.735Z] changed: [an-a01dr01]
[2026-09-29T11:07:26.735Z] 
[2026-09-29T11:07:26.735Z] TASK [verify an-test-deploy1 DR drbd clean status sync _raw_params=../../common/wait_server_drbd_dr_clean.yml] ***
[2026-09-29T11:07:26.735Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01
[2026-09-29T11:07:26.735Z] 
[2026-09-29T11:07:26.735Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:07:26.735Z] ] ***
[2026-09-29T11:08:01.745Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-29T11:08:01.746Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left).
[2026-09-29T11:08:01.746Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left).
[2026-09-29T11:08:01.746Z] ok: [an-a01dr01]
[2026-09-29T11:08:01.746Z] 
[2026-09-29T11:08:01.746Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:08:01.746Z] ] ***
[2026-09-29T11:08:01.746Z] ok: [an-a01dr01]
[2026-09-29T11:08:01.746Z] 
[2026-09-29T11:08:01.746Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed _raw_params=if [ -f /etc/drbd.d/an-test-deploy{{ servernum }}.res ]; then
[2026-09-29T11:08:01.746Z]   exit 1
[2026-09-29T11:08:01.746Z] fi
[2026-09-29T11:08:01.746Z] ] ***
[2026-09-29T11:08:30.877Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left).
[2026-09-29T11:08:30.877Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left).
[2026-09-29T11:08:30.877Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left).
[2026-09-29T11:08:30.877Z] ok: [an-a01dr01]
[2026-09-29T11:08:30.877Z] 
[2026-09-29T11:08:30.877Z] TASK [check drbd sync status on nodes after disconnect sync _raw_params=if drbdadm status an-test-deploy1 | grep -q " an-a01dr01 "; then
[2026-09-29T11:08:30.877Z]   echo "drbdadm status still reports connection to DR node (#888)"
[2026-09-29T11:08:30.877Z]   #exit 1
[2026-09-29T11:08:30.877Z] fi
[2026-09-29T11:08:30.877Z] ] ***
[2026-09-29T11:08:31.288Z] changed: [an-a01dr01 -> an-a01n01]
[2026-09-29T11:08:31.288Z] 
[2026-09-29T11:08:31.288Z] TASK [unlink DR vg from storage group _raw_params=storageuuid=$(anvil-manage-storage-groups -vv --log-secure --show | grep "Storage group" | grep "SG 1" | sed -e 's#.*\[##g' -e 's#\].*##')
[2026-09-29T11:08:31.288Z] localvguuid=$(vgs --noheadings --options vg_uuid anvil-test-vg | awk '{print $1}')
[2026-09-29T11:08:31.288Z] if [ -z "$storageuuid" ] || [ -z "$localvguuid" ]; then
[2026-09-29T11:08:31.288Z]   echo "Unable to detect storageuuid: [$storageuuid] or localvguuid: [$localvguuid]"
[2026-09-29T11:08:31.288Z]   exit 1
[2026-09-29T11:08:31.288Z] fi
[2026-09-29T11:08:31.288Z] if ! anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --group "$storageuuid" --remove --member "$localvguuid"; then
[2026-09-29T11:08:31.288Z]   echo "Unable to unlink local vg uuid to storage group"
[2026-09-29T11:08:31.288Z]   exit 1
[2026-09-29T11:08:31.288Z] fi
[2026-09-29T11:08:31.288Z] ] ***
[2026-09-29T11:08:37.927Z] changed: [an-a01dr01]
[2026-09-29T11:08:37.927Z] 
[2026-09-29T11:08:37.927Z] TASK [verify DR vg is unlinked from storage group _raw_params=if [ "$(anvil-manage-storage-groups -vv --log-secure --anvil an-anvil-01 --show | grep -A3 "Storage group" | grep anvil-test-vg | wc -l)" != 2 ]; then
[2026-09-29T11:08:37.927Z]   echo "Local vg does not appear to be unlinked from storage group (#886)"
[2026-09-29T11:08:37.927Z]   exit 1
[2026-09-29T11:08:37.927Z] fi
[2026-09-29T11:08:37.927Z] ] ***
[2026-09-29T11:08:42.434Z] changed: [an-a01dr01]
[2026-09-29T11:08:42.434Z] 
[2026-09-29T11:08:42.434Z] TASK [unlink DR host from anvil _raw_params=anvil-manage-dr --dr-host $(hostname) --unlink --anvil an-anvil-01
[2026-09-29T11:08:42.434Z] ] ***
[2026-09-29T11:08:47.837Z] changed: [an-a01dr01]
[2026-09-29T11:08:47.837Z] 
[2026-09-29T11:08:47.837Z] TASK [check no DR hosts are linked _raw_params=if ! anvil-manage-dr  -vv --log-secure --show  | grep -q "No linked DR hosts"; then
[2026-09-29T11:08:47.837Z]   echo "Detected a linked DR host!"
[2026-09-29T11:08:47.837Z]   exit 1
[2026-09-29T11:08:47.837Z] fi
[2026-09-29T11:08:47.837Z] ] ***
[2026-09-29T11:08:51.280Z] changed: [an-a01dr01]
[2026-09-29T11:08:51.280Z] 
[2026-09-29T11:08:51.280Z] PLAY [upgrade server test] *****************************************************
[2026-09-29T11:08:51.280Z] 
[2026-09-29T11:08:51.280Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:08:53.942Z] ok: [an-a01n01]
[2026-09-29T11:08:53.942Z] ok: [an-a01n02]
[2026-09-29T11:08:53.942Z] 
[2026-09-29T11:08:53.942Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T11:08:53.942Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T11:08:53.942Z] 
[2026-09-29T11:08:53.942Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T11:08:53.942Z] ] ***
[2026-09-29T11:08:57.385Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T11:08:57.385Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T11:08:57.385Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T11:08:57.385Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T11:08:57.385Z] 
[2026-09-29T11:08:57.385Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] ***
[2026-09-29T11:08:57.796Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02
[2026-09-29T11:08:57.796Z] 
[2026-09-29T11:08:57.796Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)"
[2026-09-29T11:08:57.796Z] if [ -z "$storage" ]; then
[2026-09-29T11:08:57.796Z]   echo "Unable to detect shared storage"
[2026-09-29T11:08:57.796Z]   exit 1
[2026-09-29T11:08:57.796Z] fi
[2026-09-29T11:08:57.796Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T11:08:57.796Z] ] ***
[2026-09-29T11:09:04.592Z] changed: [an-a01n02]
[2026-09-29T11:09:04.592Z] changed: [an-a01n01]
[2026-09-29T11:09:04.592Z] 
[2026-09-29T11:09:04.592Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] ***
[2026-09-29T11:09:04.592Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02
[2026-09-29T11:09:04.592Z] 
[2026-09-29T11:09:04.592Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)"
[2026-09-29T11:09:04.592Z] deployiso="$(echo "$isos" | grep "=deploy")"
[2026-09-29T11:09:04.592Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then
[2026-09-29T11:09:04.592Z]   echo "Unable to detect all deploy iso"
[2026-09-29T11:09:04.592Z]   echo "Detected isos"
[2026-09-29T11:09:04.592Z]   echo "$isos"
[2026-09-29T11:09:04.592Z]   exit 1
[2026-09-29T11:09:04.592Z] fi
[2026-09-29T11:09:04.592Z] installiso="$(echo "$isos" | grep -v "=deploy")"
[2026-09-29T11:09:04.592Z] if [ -z "$installiso" ]; then
[2026-09-29T11:09:04.592Z]   echo "Unable to detect install iso"
[2026-09-29T11:09:04.592Z]   echo "Detected isos"
[2026-09-29T11:09:04.592Z]   echo "$isos"
[2026-09-29T11:09:04.592Z]   exit 1
[2026-09-29T11:09:04.592Z] fi
[2026-09-29T11:09:04.592Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g'
[2026-09-29T11:09:04.592Z] ] ***
[2026-09-29T11:09:08.943Z] changed: [an-a01n02]
[2026-09-29T11:09:08.943Z] changed: [an-a01n01]
[2026-09-29T11:09:08.943Z] 
[2026-09-29T11:09:08.943Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] ***
[2026-09-29T11:09:08.943Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02
[2026-09-29T11:09:08.943Z] 
[2026-09-29T11:09:08.943Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:09:08.943Z] ] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] ok: [an-a01n01]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 9f187cb3-f7de-46c5-987e-3e85af53e088 _raw_params=anvil-provision-server -vv --log-secure \
[2026-09-29T11:09:10.927Z]   --ci-test \
[2026-09-29T11:09:10.927Z]   --name an-test-deploy{{ servernum }} \
[2026-09-29T11:09:10.927Z]   --os almalinux9 \
[2026-09-29T11:09:10.927Z]   --cpu 2 --ram 4G \
[2026-09-29T11:09:10.927Z]   --storage-group {{ storageuuid }} --storage-size 10G \
[2026-09-29T11:09:10.927Z]   --install-media {{ installisouuid }} \
[2026-09-29T11:09:10.927Z]   --driver-disc deploy{{ servernum }}.iso
[2026-09-29T11:09:10.927Z] ] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n01]
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n01]
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n01]
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n01]
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] ***
[2026-09-29T11:09:10.927Z] skipping: [an-a01n01]
[2026-09-29T11:09:10.927Z] skipping: [an-a01n02]
[2026-09-29T11:09:10.927Z] 
[2026-09-29T11:09:10.927Z] TASK [check servers status _raw_params=../../common/server_started.yml] ********
[2026-09-29T11:09:11.337Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02
[2026-09-29T11:09:11.337Z] 
[2026-09-29T11:09:11.337Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T11:09:11.337Z] numservers="$(echo "$pcsstatus" | wc -l)"
[2026-09-29T11:09:11.337Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)"
[2026-09-29T11:09:11.337Z] if [ "$numservers" != "$startedservers" ]; then
[2026-09-29T11:09:11.337Z]   echo "Detected $numservers but only $startedservers in Started state"
[2026-09-29T11:09:11.337Z]   echo "pcs:"
[2026-09-29T11:09:11.337Z]   echo "$pcsstatus"
[2026-09-29T11:09:11.337Z]   exit 1
[2026-09-29T11:09:11.337Z] fi
[2026-09-29T11:09:11.337Z] exit 0
[2026-09-29T11:09:11.337Z] ] ***
[2026-09-29T11:09:13.321Z] ok: [an-a01n02]
[2026-09-29T11:09:13.321Z] ok: [an-a01n01]
[2026-09-29T11:09:13.321Z] 
[2026-09-29T11:09:13.321Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)"
[2026-09-29T11:09:13.321Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)"
[2026-09-29T11:09:13.321Z] servers="$(echo "$hypervisor" | wc -l)"
[2026-09-29T11:09:13.321Z] ret=0
[2026-09-29T11:09:13.321Z] if [ "$servers"  != "1" ]; then
[2026-09-29T11:09:13.321Z]   echo "Not all servers are running on the same node!"
[2026-09-29T11:09:13.321Z]   echo "$pcsstatus"
[2026-09-29T11:09:13.321Z]   ret=1
[2026-09-29T11:09:13.321Z] fi
[2026-09-29T11:09:13.321Z] echo "$hypervisor"
[2026-09-29T11:09:13.321Z] exit $ret
[2026-09-29T11:09:13.321Z] ] ***
[2026-09-29T11:09:15.305Z] ok: [an-a01n01]
[2026-09-29T11:09:15.305Z] ok: [an-a01n02]
[2026-09-29T11:09:15.305Z] 
[2026-09-29T11:09:15.305Z] PLAY [enable upgrade repo] *****************************************************
[2026-09-29T11:09:15.305Z] 
[2026-09-29T11:09:15.305Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:09:17.968Z] ok: [an-a01dr01]
[2026-09-29T11:09:17.968Z] ok: [an-a01n02]
[2026-09-29T11:09:17.968Z] ok: [an-a01n01]
[2026-09-29T11:09:17.968Z] ok: [an-striker02]
[2026-09-29T11:09:17.968Z] ok: [an-striker01]
[2026-09-29T11:09:17.968Z] 
[2026-09-29T11:09:17.968Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}')
[2026-09-29T11:09:17.968Z] dnf config-manager --set-enabled $updaterepo
[2026-09-29T11:09:17.968Z] ] ***
[2026-09-29T11:09:29.926Z] changed: [an-a01n02]
[2026-09-29T11:09:29.926Z] changed: [an-a01dr01]
[2026-09-29T11:09:29.926Z] changed: [an-a01n01]
[2026-09-29T11:09:29.926Z] changed: [an-striker02]
[2026-09-29T11:09:29.926Z] changed: [an-striker01]
[2026-09-29T11:09:29.926Z] 
[2026-09-29T11:09:29.926Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] ***
[2026-09-29T11:09:29.926Z] ok: [an-striker01]
[2026-09-29T11:09:29.926Z] ok: [an-striker02]
[2026-09-29T11:09:30.337Z] ok: [an-a01n01]
[2026-09-29T11:09:30.337Z] ok: [an-a01n02]
[2026-09-29T11:09:30.337Z] ok: [an-a01dr01]
[2026-09-29T11:09:30.337Z] 
[2026-09-29T11:09:30.337Z] PLAY [start upgrade process] ***************************************************
[2026-09-29T11:09:30.337Z] 
[2026-09-29T11:09:30.337Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:09:32.323Z] ok: [an-striker01]
[2026-09-29T11:09:32.323Z] 
[2026-09-29T11:09:32.323Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes
[2026-09-29T11:09:32.323Z] ] ***
[2026-09-29T11:41:47.770Z] changed: [an-striker01]
[2026-09-29T11:41:47.770Z] 
[2026-09-29T11:41:47.770Z] TASK [reboot striker01 reboot_timeout=1800] ************************************
[2026-09-29T11:42:22.582Z] changed: [an-striker01]
[2026-09-29T11:42:22.582Z] 
[2026-09-29T11:42:22.582Z] PLAY [reconnect all nodes] *****************************************************
[2026-09-29T11:42:22.582Z] 
[2026-09-29T11:42:22.582Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:42:22.582Z] ok: [an-a01n02]
[2026-09-29T11:42:22.582Z] ok: [an-a01dr01]
[2026-09-29T11:42:22.582Z] ok: [an-a01n01]
[2026-09-29T11:42:22.582Z] ok: [an-striker02]
[2026-09-29T11:42:22.582Z] ok: [an-striker01]
[2026-09-29T11:42:22.582Z] 
[2026-09-29T11:42:22.582Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] ***
[2026-09-29T11:42:22.582Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01
[2026-09-29T11:42:22.582Z] 
[2026-09-29T11:42:22.582Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] ***
[2026-09-29T11:45:33.678Z] ok: [an-striker01 -> localhost]
[2026-09-29T11:45:33.678Z] ok: [an-striker02 -> localhost]
[2026-09-29T11:45:33.678Z] ok: [an-a01n01 -> localhost]
[2026-09-29T11:45:33.678Z] ok: [an-a01n02 -> localhost]
[2026-09-29T11:45:33.678Z] ok: [an-a01dr01 -> localhost]
[2026-09-29T11:45:33.678Z] 
[2026-09-29T11:45:33.678Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then
[2026-09-29T11:45:33.678Z]   echo "Node current uptime higher than expected! tool failed to reboot nodes?"
[2026-09-29T11:45:33.678Z]   exit 1
[2026-09-29T11:45:33.678Z] fi
[2026-09-29T11:45:33.678Z] ] ***
[2026-09-29T11:45:35.069Z] changed: [an-a01n02]
[2026-09-29T11:45:35.069Z] changed: [an-a01n01]
[2026-09-29T11:45:35.069Z] changed: [an-striker01]
[2026-09-29T11:45:35.069Z] changed: [an-striker02]
[2026-09-29T11:45:35.069Z] changed: [an-a01dr01]
[2026-09-29T11:45:35.069Z] 
[2026-09-29T11:45:35.069Z] PLAY [check all nodes status after upgrade] ************************************
[2026-09-29T11:45:35.069Z] 
[2026-09-29T11:45:35.069Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:45:37.890Z] ok: [an-a01dr01]
[2026-09-29T11:45:37.890Z] ok: [an-a01n02]
[2026-09-29T11:45:37.890Z] ok: [an-a01n01]
[2026-09-29T11:45:37.890Z] ok: [an-striker01]
[2026-09-29T11:45:37.890Z] ok: [an-striker02]
[2026-09-29T11:45:37.890Z] 
[2026-09-29T11:45:37.890Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then
[2026-09-29T11:45:37.890Z]   echo "perl-Log-Journald update not found!"
[2026-09-29T11:45:37.890Z]   exit 1
[2026-09-29T11:45:37.890Z] fi
[2026-09-29T11:45:37.890Z] ] ***
[2026-09-29T11:45:41.334Z] changed: [an-a01dr01]
[2026-09-29T11:45:41.334Z] changed: [an-a01n01]
[2026-09-29T11:45:41.334Z] changed: [an-a01n02]
[2026-09-29T11:45:41.334Z] changed: [an-striker01]
[2026-09-29T11:45:41.334Z] changed: [an-striker02]
[2026-09-29T11:45:41.334Z] 
[2026-09-29T11:45:41.334Z] PLAY [check nodes status after upgrade] ****************************************
[2026-09-29T11:45:41.334Z] 
[2026-09-29T11:45:41.334Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:45:42.726Z] ok: [an-a01dr01]
[2026-09-29T11:45:42.726Z] ok: [an-a01n02]
[2026-09-29T11:45:42.726Z] ok: [an-a01n01]
[2026-09-29T11:45:42.726Z] 
[2026-09-29T11:45:42.726Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then
[2026-09-29T11:45:42.726Z]   echo "akmod-drbd update not found!"
[2026-09-29T11:45:42.726Z]   exit 1
[2026-09-29T11:45:42.726Z] fi
[2026-09-29T11:45:42.726Z] ] ***
[2026-09-29T11:45:44.711Z] changed: [an-a01n02]
[2026-09-29T11:45:44.711Z] changed: [an-a01n01]
[2026-09-29T11:45:44.711Z] changed: [an-a01dr01]
[2026-09-29T11:45:44.711Z] 
[2026-09-29T11:45:44.711Z] PLAY [check hosted node status after upgrade] **********************************
[2026-09-29T11:45:44.711Z] 
[2026-09-29T11:45:44.711Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:45:46.696Z] ok: [an-a01n02]
[2026-09-29T11:45:46.696Z] ok: [an-a01n01]
[2026-09-29T11:45:46.696Z] 
[2026-09-29T11:45:46.696Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T11:45:46.696Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T11:45:46.696Z] 
[2026-09-29T11:45:46.696Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T11:45:46.696Z] ] ***
[2026-09-29T11:45:51.051Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T11:45:51.051Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T11:45:51.051Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T11:45:51.051Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T11:45:51.051Z] 
[2026-09-29T11:45:51.051Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] ***
[2026-09-29T11:45:51.052Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02
[2026-09-29T11:45:51.052Z] 
[2026-09-29T11:45:51.052Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started
[2026-09-29T11:45:51.052Z] ] ***
[2026-09-29T11:45:52.446Z] ok: [an-a01n02]
[2026-09-29T11:45:52.446Z] ok: [an-a01n01]
[2026-09-29T11:45:52.446Z] 
[2026-09-29T11:45:52.446Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] ***
[2026-09-29T11:45:52.446Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02
[2026-09-29T11:45:52.446Z] 
[2026-09-29T11:45:52.446Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l
[2026-09-29T11:45:52.446Z] ] ***
[2026-09-29T11:45:52.857Z] ok: [an-a01n01]
[2026-09-29T11:45:52.857Z] ok: [an-a01n02]
[2026-09-29T11:45:52.857Z] 
[2026-09-29T11:45:52.857Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] ***
[2026-09-29T11:45:53.268Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02
[2026-09-29T11:45:53.268Z] 
[2026-09-29T11:45:53.268Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }}
[2026-09-29T11:45:53.268Z] ] ***
[2026-09-29T11:45:54.298Z] ok: [an-a01n01]
[2026-09-29T11:45:54.298Z] ok: [an-a01n02]
[2026-09-29T11:45:54.298Z] 
[2026-09-29T11:45:54.298Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] ***
[2026-09-29T11:45:54.298Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02
[2026-09-29T11:45:54.298Z] 
[2026-09-29T11:45:54.298Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22
[2026-09-29T11:45:54.298Z] ] ***
[2026-09-29T11:45:55.170Z] ok: [an-a01n01]
[2026-09-29T11:45:55.170Z] ok: [an-a01n02]
[2026-09-29T11:45:55.170Z] 
[2026-09-29T11:45:55.170Z] PLAY [test anvil package self update] ******************************************
[2026-09-29T11:45:55.170Z] 
[2026-09-29T11:45:55.170Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:45:58.616Z] ok: [an-a01n02]
[2026-09-29T11:45:58.616Z] ok: [an-a01dr01]
[2026-09-29T11:45:58.616Z] ok: [an-a01n01]
[2026-09-29T11:45:58.616Z] ok: [an-striker01]
[2026-09-29T11:45:58.616Z] ok: [an-striker02]
[2026-09-29T11:45:58.616Z] 
[2026-09-29T11:45:58.616Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] ***
[2026-09-29T11:46:02.060Z] changed: [an-a01n02] => (item=anvil-daemon.service)
[2026-09-29T11:46:02.060Z] changed: [an-a01n01] => (item=anvil-daemon.service)
[2026-09-29T11:46:02.060Z] changed: [an-a01dr01] => (item=anvil-daemon.service)
[2026-09-29T11:46:02.060Z] changed: [an-striker01] => (item=anvil-daemon.service)
[2026-09-29T11:46:02.060Z] changed: [an-striker02] => (item=anvil-daemon.service)
[2026-09-29T11:46:02.060Z] changed: [an-a01n02] => (item=scancore.service)
[2026-09-29T11:46:02.060Z] changed: [an-a01dr01] => (item=scancore.service)
[2026-09-29T11:46:02.060Z] changed: [an-a01n01] => (item=scancore.service)
[2026-09-29T11:46:02.060Z] changed: [an-striker01] => (item=scancore.service)
[2026-09-29T11:46:02.060Z] changed: [an-striker02] => (item=scancore.service)
[2026-09-29T11:46:02.060Z] 
[2026-09-29T11:46:02.060Z] PLAY [test striker package self update] ****************************************
[2026-09-29T11:46:02.060Z] 
[2026-09-29T11:46:02.060Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:46:02.931Z] ok: [an-striker01]
[2026-09-29T11:46:02.931Z] ok: [an-striker02]
[2026-09-29T11:46:02.931Z] 
[2026-09-29T11:46:02.931Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker
[2026-09-29T11:46:02.931Z] ] ***
[2026-09-29T11:46:32.057Z] changed: [an-striker01]
[2026-09-29T11:46:32.057Z] changed: [an-striker02]
[2026-09-29T11:46:32.057Z] 
[2026-09-29T11:46:32.057Z] PLAY [test node package self update] *******************************************
[2026-09-29T11:46:32.057Z] 
[2026-09-29T11:46:32.057Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:46:33.606Z] ok: [an-a01n02]
[2026-09-29T11:46:33.606Z] ok: [an-a01n01]
[2026-09-29T11:46:33.606Z] 
[2026-09-29T11:46:33.606Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node
[2026-09-29T11:46:33.606Z] ] ***
[2026-09-29T11:47:02.737Z] changed: [an-a01n02]
[2026-09-29T11:47:02.737Z] changed: [an-a01n01]
[2026-09-29T11:47:02.737Z] 
[2026-09-29T11:47:02.737Z] PLAY [test dr package self update] *********************************************
[2026-09-29T11:47:02.737Z] 
[2026-09-29T11:47:02.737Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:47:03.607Z] ok: [an-a01dr01]
[2026-09-29T11:47:03.607Z] 
[2026-09-29T11:47:03.607Z] TASK [reinstall anvil-core and anvil-dr _raw_params=yum reinstall -y anvil-core anvil-dr
[2026-09-29T11:47:03.607Z] ] ***
[2026-09-29T11:47:32.892Z] changed: [an-a01dr01]
[2026-09-29T11:47:32.892Z] 
[2026-09-29T11:47:32.892Z] PLAY [test anvil package self update] ******************************************
[2026-09-29T11:47:32.892Z] 
[2026-09-29T11:47:32.892Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:47:32.892Z] ok: [an-striker01]
[2026-09-29T11:47:32.892Z] ok: [an-a01n02]
[2026-09-29T11:47:32.892Z] ok: [an-striker02]
[2026-09-29T11:47:32.892Z] ok: [an-a01dr01]
[2026-09-29T11:47:32.892Z] ok: [an-a01n01]
[2026-09-29T11:47:32.892Z] 
[2026-09-29T11:47:32.892Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] ***
[2026-09-29T11:47:33.302Z] changed: [an-a01dr01] => (item=anvil-daemon.service)
[2026-09-29T11:47:33.302Z] changed: [an-a01n02] => (item=anvil-daemon.service)
[2026-09-29T11:47:33.302Z] changed: [an-a01n01] => (item=anvil-daemon.service)
[2026-09-29T11:47:33.302Z] changed: [an-striker01] => (item=anvil-daemon.service)
[2026-09-29T11:47:33.302Z] changed: [an-striker02] => (item=anvil-daemon.service)
[2026-09-29T11:47:33.302Z] changed: [an-a01dr01] => (item=scancore.service)
[2026-09-29T11:47:33.302Z] changed: [an-a01n02] => (item=scancore.service)
[2026-09-29T11:47:33.302Z] changed: [an-a01n01] => (item=scancore.service)
[2026-09-29T11:47:33.302Z] changed: [an-striker01] => (item=scancore.service)
[2026-09-29T11:47:33.302Z] changed: [an-striker02] => (item=scancore.service)
[2026-09-29T11:47:33.302Z] 
[2026-09-29T11:47:33.302Z] PLAY [delete server tests] *****************************************************
[2026-09-29T11:47:33.302Z] 
[2026-09-29T11:47:33.302Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:47:35.286Z] ok: [an-a01n02]
[2026-09-29T11:47:35.286Z] ok: [an-a01n01]
[2026-09-29T11:47:35.286Z] 
[2026-09-29T11:47:35.286Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] ***
[2026-09-29T11:47:35.286Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02
[2026-09-29T11:47:35.286Z] 
[2026-09-29T11:47:35.286Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}"
[2026-09-29T11:47:35.286Z] ] ***
[2026-09-29T11:47:39.637Z] ok: [an-a01n02] => (item=an-a01n01)
[2026-09-29T11:47:39.637Z] ok: [an-a01n01] => (item=an-a01n01)
[2026-09-29T11:47:39.637Z] ok: [an-a01n02] => (item=an-a01n02)
[2026-09-29T11:47:39.637Z] ok: [an-a01n01] => (item=an-a01n02)
[2026-09-29T11:47:39.637Z] 
[2026-09-29T11:47:39.637Z] TASK [delete servers _raw_params=../../common/delete_server.yml] ***************
[2026-09-29T11:47:39.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1)
[2026-09-29T11:47:39.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2)
[2026-09-29T11:47:39.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3)
[2026-09-29T11:47:39.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4)
[2026-09-29T11:47:39.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5)
[2026-09-29T11:47:39.637Z] 
[2026-09-29T11:47:39.637Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:47:39.637Z] ] ***
[2026-09-29T11:47:41.030Z] skipping: [an-a01n01]
[2026-09-29T11:47:41.030Z] ok: [an-a01n02]
[2026-09-29T11:47:41.030Z] 
[2026-09-29T11:47:41.030Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T11:47:41.030Z]   --force \
[2026-09-29T11:47:41.030Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T11:47:41.030Z] ] ***
[2026-09-29T11:47:49.115Z] skipping: [an-a01n01]
[2026-09-29T11:47:49.115Z] changed: [an-a01n02]
[2026-09-29T11:47:49.115Z] 
[2026-09-29T11:47:49.115Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:47:49.115Z] ] ***
[2026-09-29T11:48:38.881Z] skipping: [an-a01n01]
[2026-09-29T11:48:38.881Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left).
[2026-09-29T11:48:38.881Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left).
[2026-09-29T11:48:38.881Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left).
[2026-09-29T11:48:38.881Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left).
[2026-09-29T11:48:38.881Z] ok: [an-a01n02]
[2026-09-29T11:48:38.881Z] 
[2026-09-29T11:48:38.881Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:48:38.881Z] ] ***
[2026-09-29T11:48:48.668Z] skipping: [an-a01n01]
[2026-09-29T11:48:48.668Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left).
[2026-09-29T11:48:48.668Z] ok: [an-a01n02]
[2026-09-29T11:48:48.668Z] 
[2026-09-29T11:48:48.668Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:48:48.668Z] ] ***
[2026-09-29T11:48:49.540Z] skipping: [an-a01n01]
[2026-09-29T11:48:49.540Z] ok: [an-a01n02]
[2026-09-29T11:48:49.540Z] 
[2026-09-29T11:48:49.540Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:49.540Z] ] ***
[2026-09-29T11:48:50.931Z] skipping: [an-a01n01]
[2026-09-29T11:48:50.931Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.923988", "end": "2026-09-29 07:48:50.614718", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 07:48:49.690730", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T11:48:50.931Z] ...ignoring
[2026-09-29T11:48:50.931Z] 
[2026-09-29T11:48:50.931Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T11:48:50.931Z]   --force \
[2026-09-29T11:48:50.931Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T11:48:50.931Z] ] ***
[2026-09-29T11:48:50.931Z] skipping: [an-a01n01]
[2026-09-29T11:48:50.931Z] skipping: [an-a01n02]
[2026-09-29T11:48:50.931Z] 
[2026-09-29T11:48:50.931Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:50.931Z] ] ***
[2026-09-29T11:48:50.931Z] skipping: [an-a01n01]
[2026-09-29T11:48:50.931Z] skipping: [an-a01n02]
[2026-09-29T11:48:50.931Z] 
[2026-09-29T11:48:50.931Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:48:50.931Z] ] ***
[2026-09-29T11:48:51.342Z] skipping: [an-a01n01]
[2026-09-29T11:48:51.342Z] skipping: [an-a01n02]
[2026-09-29T11:48:51.342Z] 
[2026-09-29T11:48:51.342Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:48:51.342Z] ] ***
[2026-09-29T11:48:51.342Z] skipping: [an-a01n01]
[2026-09-29T11:48:51.342Z] skipping: [an-a01n02]
[2026-09-29T11:48:51.342Z] 
[2026-09-29T11:48:51.342Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:51.342Z] ] ***
[2026-09-29T11:48:52.740Z] skipping: [an-a01n01]
[2026-09-29T11:48:52.740Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.926844", "end": "2026-09-29 07:48:52.493858", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 07:48:51.567014", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T11:48:52.740Z] ...ignoring
[2026-09-29T11:48:52.740Z] 
[2026-09-29T11:48:52.740Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T11:48:52.740Z]   --force \
[2026-09-29T11:48:52.740Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T11:48:52.740Z] ] ***
[2026-09-29T11:48:52.740Z] skipping: [an-a01n01]
[2026-09-29T11:48:52.740Z] skipping: [an-a01n02]
[2026-09-29T11:48:52.740Z] 
[2026-09-29T11:48:52.740Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:52.740Z] ] ***
[2026-09-29T11:48:52.740Z] skipping: [an-a01n01]
[2026-09-29T11:48:52.740Z] skipping: [an-a01n02]
[2026-09-29T11:48:52.740Z] 
[2026-09-29T11:48:52.740Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:48:52.740Z] ] ***
[2026-09-29T11:48:53.150Z] skipping: [an-a01n01]
[2026-09-29T11:48:53.150Z] skipping: [an-a01n02]
[2026-09-29T11:48:53.150Z] 
[2026-09-29T11:48:53.150Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:48:53.150Z] ] ***
[2026-09-29T11:48:53.150Z] skipping: [an-a01n01]
[2026-09-29T11:48:53.150Z] skipping: [an-a01n02]
[2026-09-29T11:48:53.150Z] 
[2026-09-29T11:48:53.150Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:53.150Z] ] ***
[2026-09-29T11:48:54.698Z] skipping: [an-a01n01]
[2026-09-29T11:48:54.698Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.937109", "end": "2026-09-29 07:48:54.359691", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 07:48:53.422582", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T11:48:54.698Z] ...ignoring
[2026-09-29T11:48:54.698Z] 
[2026-09-29T11:48:54.698Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T11:48:54.698Z]   --force \
[2026-09-29T11:48:54.698Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T11:48:54.698Z] ] ***
[2026-09-29T11:48:54.698Z] skipping: [an-a01n01]
[2026-09-29T11:48:54.698Z] skipping: [an-a01n02]
[2026-09-29T11:48:54.698Z] 
[2026-09-29T11:48:54.698Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:54.698Z] ] ***
[2026-09-29T11:48:54.698Z] skipping: [an-a01n01]
[2026-09-29T11:48:54.698Z] skipping: [an-a01n02]
[2026-09-29T11:48:54.698Z] 
[2026-09-29T11:48:54.698Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:48:54.698Z] ] ***
[2026-09-29T11:48:54.698Z] skipping: [an-a01n01]
[2026-09-29T11:48:54.698Z] skipping: [an-a01n02]
[2026-09-29T11:48:54.698Z] 
[2026-09-29T11:48:54.698Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:48:54.698Z] ] ***
[2026-09-29T11:48:55.108Z] skipping: [an-a01n01]
[2026-09-29T11:48:55.108Z] skipping: [an-a01n02]
[2026-09-29T11:48:55.108Z] 
[2026-09-29T11:48:55.108Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:55.108Z] ] ***
[2026-09-29T11:48:56.500Z] skipping: [an-a01n01]
[2026-09-29T11:48:56.500Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.916807", "end": "2026-09-29 07:48:56.173662", "msg": "non-zero return code", "rc": 1, "start": "2026-09-29 07:48:55.256855", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []}
[2026-09-29T11:48:56.500Z] ...ignoring
[2026-09-29T11:48:56.500Z] 
[2026-09-29T11:48:56.500Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \
[2026-09-29T11:48:56.500Z]   --force \
[2026-09-29T11:48:56.500Z]   --server an-test-deploy{{ servernum }}
[2026-09-29T11:48:56.500Z] ] ***
[2026-09-29T11:48:56.500Z] skipping: [an-a01n01]
[2026-09-29T11:48:56.500Z] skipping: [an-a01n02]
[2026-09-29T11:48:56.500Z] 
[2026-09-29T11:48:56.500Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }}
[2026-09-29T11:48:56.500Z] ] ***
[2026-09-29T11:48:56.500Z] skipping: [an-a01n01]
[2026-09-29T11:48:56.500Z] skipping: [an-a01n02]
[2026-09-29T11:48:56.500Z] 
[2026-09-29T11:48:56.500Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }}
[2026-09-29T11:48:56.500Z] ] ***
[2026-09-29T11:48:56.500Z] skipping: [an-a01n01]
[2026-09-29T11:48:56.500Z] skipping: [an-a01n02]
[2026-09-29T11:48:56.500Z] 
[2026-09-29T11:48:56.500Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }}
[2026-09-29T11:48:56.500Z] ] ***
[2026-09-29T11:48:56.910Z] skipping: [an-a01n01]
[2026-09-29T11:48:56.910Z] skipping: [an-a01n02]
[2026-09-29T11:48:56.910Z] 
[2026-09-29T11:48:56.910Z] PLAY [Check anvil available resources from striker01] **************************
[2026-09-29T11:48:56.910Z] 
[2026-09-29T11:48:56.910Z] TASK [Gathering Facts ] ********************************************************
[2026-09-29T11:48:58.893Z] ok: [an-striker01]
[2026-09-29T11:48:58.893Z] 
[2026-09-29T11:48:58.893Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status"
[2026-09-29T11:48:58.893Z] sleep 300
[2026-09-29T11:48:58.893Z] anvil-report-usage > /root/anvil_available_resources.stop
[2026-09-29T11:48:58.893Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop
[2026-09-29T11:48:58.893Z] ] ***
[2026-09-29T11:54:11.458Z] ok: [an-striker01]
[2026-09-29T11:54:11.458Z] 
[2026-09-29T11:54:11.458Z] PLAY RECAP *********************************************************************
[2026-09-29T11:54:11.458Z] an-a01dr01                 : ok=90   changed=38   unreachable=0    failed=0    skipped=5    rescued=0    ignored=0   
[2026-09-29T11:54:11.458Z] an-a01n01                  : ok=314  changed=28   unreachable=0    failed=0    skipped=109  rescued=0    ignored=5   
[2026-09-29T11:54:11.458Z] an-a01n02                  : ok=345  changed=27   unreachable=0    failed=0    skipped=90   rescued=0    ignored=4   
[2026-09-29T11:54:11.458Z] an-striker01               : ok=57   changed=19   unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
[2026-09-29T11:54:11.458Z] an-striker02               : ok=42   changed=12   unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
[2026-09-29T11:54:11.458Z] 
[2026-09-29T11:54:11.458Z] tar: Removing leading `/' from member names
[2026-09-29T11:54:11.458Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts.
[2026-09-29T11:54:11.458Z] /root/ci-tools
[2026-09-29T11:54:11.458Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/1280.tar.gz
[2026-09-29T11:54:11.458Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat.
[2026-09-29T11:54:11.458Z] This might take a few seconds.
[2026-09-29T11:54:11.458Z] 
[2026-09-29T11:54:11.458Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-29T11:54:11.458Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-29T11:54:11.458Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-29T11:54:11.458Z] 
[2026-09-29T11:54:11.458Z] Manage your connected systems: https://red.ht/connector
[2026-09-29T11:54:11.869Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-29T11:54:11.869Z] 
[2026-09-29T11:54:11.869Z] 1. Register with subscription-manager
[2026-09-29T11:54:11.869Z] # subscription-manager register
[2026-09-29T11:54:11.869Z] 
[2026-09-29T11:54:11.869Z] 2. Register with insights-client
[2026-09-29T11:54:11.869Z] # insights-client --register
[2026-09-29T11:54:12.741Z] This system is currently not registered.
[2026-09-29T11:54:13.612Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat.
[2026-09-29T11:54:13.612Z] This might take a few seconds.
[2026-09-29T11:54:13.612Z] 
[2026-09-29T11:54:13.612Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-29T11:54:17.059Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-29T11:54:18.451Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-29T11:54:18.451Z] 
[2026-09-29T11:54:18.451Z] Manage your connected systems: https://red.ht/connector
[2026-09-29T11:54:21.115Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-29T11:54:21.115Z] 
[2026-09-29T11:54:21.115Z] 1. Register with subscription-manager
[2026-09-29T11:54:21.115Z] # subscription-manager register
[2026-09-29T11:54:21.115Z] 
[2026-09-29T11:54:21.115Z] 2. Register with insights-client
[2026-09-29T11:54:21.115Z] # insights-client --register
[2026-09-29T11:54:22.509Z] This system is currently not registered.
[2026-09-29T11:54:22.919Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat.
[2026-09-29T11:54:22.919Z] This might take a few seconds.
[2026-09-29T11:54:22.919Z] 
[2026-09-29T11:54:23.330Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-29T11:54:25.314Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-29T11:54:26.183Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-29T11:54:26.183Z] 
[2026-09-29T11:54:26.183Z] Manage your connected systems: https://red.ht/connector
[2026-09-29T11:54:28.166Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-29T11:54:28.166Z] 
[2026-09-29T11:54:28.166Z] 1. Register with subscription-manager
[2026-09-29T11:54:28.166Z] # subscription-manager register
[2026-09-29T11:54:28.166Z] 
[2026-09-29T11:54:28.166Z] 2. Register with insights-client
[2026-09-29T11:54:28.166Z] # insights-client --register
[2026-09-29T11:54:28.577Z] This system is currently not registered.
[2026-09-29T11:54:28.988Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat.
[2026-09-29T11:54:28.988Z] This might take a few seconds.
[2026-09-29T11:54:28.988Z] 
[2026-09-29T11:54:28.988Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-29T11:54:31.652Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-29T11:54:32.063Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-29T11:54:32.063Z] 
[2026-09-29T11:54:32.063Z] Manage your connected systems: https://red.ht/connector
[2026-09-29T11:54:34.046Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-29T11:54:34.046Z] 
[2026-09-29T11:54:34.046Z] 1. Register with subscription-manager
[2026-09-29T11:54:34.046Z] # subscription-manager register
[2026-09-29T11:54:34.046Z] 
[2026-09-29T11:54:34.046Z] 2. Register with insights-client
[2026-09-29T11:54:34.046Z] # insights-client --register
[2026-09-29T11:54:34.918Z] This system is currently not registered.
[2026-09-29T11:54:35.328Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat.
[2026-09-29T11:54:35.328Z] This might take a few seconds.
[2026-09-29T11:54:35.328Z] 
[2026-09-29T11:54:35.328Z] 𐄂 Deactivated the Remote Host Configuration daemon
[2026-09-29T11:54:37.311Z] 𐄂 Disconnected from Red Hat Insights
[2026-09-29T11:54:38.182Z] 𐄂 Disconnected from Red Hat Subscription Management
[2026-09-29T11:54:38.182Z] 
[2026-09-29T11:54:38.182Z] Manage your connected systems: https://red.ht/connector
[2026-09-29T11:54:39.728Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client.
[2026-09-29T11:54:39.728Z] 
[2026-09-29T11:54:39.728Z] 1. Register with subscription-manager
[2026-09-29T11:54:39.728Z] # subscription-manager register
[2026-09-29T11:54:39.728Z] 
[2026-09-29T11:54:39.728Z] 2. Register with insights-client
[2026-09-29T11:54:39.728Z] # insights-client --register
[2026-09-29T11:54:41.122Z] This system is currently not registered.
[2026-09-29T11:54:41.243Z] [Pipeline] }
[2026-09-29T11:54:41.246Z] [Pipeline] // timeout
[2026-09-29T11:54:41.249Z] [Pipeline] }
[2026-09-29T11:54:41.251Z] [Pipeline] // script
[2026-09-29T11:54:41.254Z] [Pipeline] }
[2026-09-29T11:54:41.256Z] [Pipeline] // stage
[2026-09-29T11:54:41.259Z] [Pipeline] stage
[2026-09-29T11:54:41.259Z] [Pipeline] { (Declarative: Post Actions)
[2026-09-29T11:54:41.264Z] [Pipeline] cleanWs
[2026-09-29T11:54:41.347Z] [WS-CLEANUP] Deleting project workspace...
[2026-09-29T11:54:41.347Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[2026-09-29T11:54:41.388Z] [WS-CLEANUP] done
[2026-09-29T11:54:41.392Z] [Pipeline] script
[2026-09-29T11:54:41.393Z] [Pipeline] {
[2026-09-29T11:54:41.407Z] [Pipeline] stage
[2026-09-29T11:54:41.408Z] [Pipeline] { (Copying logs)
[2026-09-29T11:54:41.440Z] [Pipeline] node
[2026-09-29T11:54:41.540Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing
[2026-09-29T11:54:41.541Z] [Pipeline] {
[2026-09-29T11:54:41.581Z] [Pipeline] node
[2026-09-29T11:54:41.679Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2
[2026-09-29T11:54:41.679Z] [Pipeline] {
[2026-09-29T11:54:41.683Z] [Pipeline] sh
[2026-09-29T11:54:41.938Z] + mkdir -p /var/lib/jenkins/locks
[2026-09-29T11:54:41.947Z] [Pipeline] echo
[2026-09-29T11:54:41.947Z] RWLock: FD for lock log_archive in stage projectFinishUp is 584
[2026-09-29T11:54:41.950Z] [Pipeline] sleep
[2026-09-29T11:54:41.950Z] No need to sleep any longer
[2026-09-29T11:54:41.953Z] [Pipeline] echo
[2026-09-29T11:54:41.953Z] RWLock: log_archive in stage projectFinishUp locked for WRITE
[2026-09-29T11:54:41.956Z] [Pipeline] }
[2026-09-29T11:54:41.958Z] [Pipeline] // node
[2026-09-29T11:54:41.963Z] [Pipeline] echo
[2026-09-29T11:54:41.963Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1280/
[2026-09-29T11:54:41.964Z] [Pipeline] sh
[2026-09-29T11:54:42.219Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1280/
[2026-09-29T11:54:42.220Z] [Pipeline] sh