[2026-05-08T08:47:07.517Z] Started by upstream project "anvil/anvil-build-cron" build number 48 [2026-05-08T08:47:07.517Z] originally caused by: [2026-05-08T08:47:07.517Z] Started by timer [2026-05-08T08:47:08.469Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-05-08T08:47:08.477Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-05-08T08:47:08.673Z] Examining kronosnet/ci-tools [2026-05-08T08:47:08.673Z] Attempting to resolve main as a branch [2026-05-08T08:47:08.836Z] Resolved main as branch main at revision ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f [2026-05-08T08:47:08.837Z] Selected Git installation does not exist. Using Default [2026-05-08T08:47:08.837Z] The recommended git tool is: NONE [2026-05-08T08:47:08.837Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-05-08T08:47:08.841Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-05-08T08:47:08.841Z] Wiping out workspace first. [2026-05-08T08:47:08.862Z] Cloning the remote Git repository [2026-05-08T08:47:08.862Z] Using shallow clone with depth 1 [2026-05-08T08:47:08.862Z] Avoid fetching tags [2026-05-08T08:47:08.862Z] Honoring refspec on initial clone [2026-05-08T08:47:08.862Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-05-08T08:47:08.863Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10 [2026-05-08T08:47:08.874Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-05-08T08:47:08.874Z] > git --version # timeout=10 [2026-05-08T08:47:08.881Z] > git --version # 'git version 2.47.3' [2026-05-08T08:47:08.881Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-05-08T08:47:08.888Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-08T08:47:09.124Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-05-08T08:47:09.133Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-08T08:47:09.144Z] Avoid second fetch [2026-05-08T08:47:09.144Z] Checking out Revision ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f (main) [2026-05-08T08:47:09.144Z] > git config core.sparsecheckout # timeout=10 [2026-05-08T08:47:09.151Z] > git checkout -f ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f # timeout=10 [2026-05-08T08:47:09.184Z] Commit message: "fn-testing: add allprio= to osp" [2026-05-08T08:47:09.190Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-05-08T08:47:09.433Z] Examining kronosnet/ci-tools [2026-05-08T08:47:09.434Z] Attempting to resolve main as a branch [2026-05-08T08:47:09.638Z] Resolved main as branch main at revision ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f [2026-05-08T08:47:09.639Z] Selected Git installation does not exist. Using Default [2026-05-08T08:47:09.639Z] The recommended git tool is: NONE [2026-05-08T08:47:09.639Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-05-08T08:47:09.642Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-05-08T08:47:09.642Z] Wiping out workspace first. [2026-05-08T08:47:09.660Z] Cloning the remote Git repository [2026-05-08T08:47:09.660Z] Using shallow clone with depth 1 [2026-05-08T08:47:09.660Z] Avoid fetching tags [2026-05-08T08:47:09.660Z] Honoring refspec on initial clone [2026-05-08T08:47:09.660Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-05-08T08:47:09.660Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@4 # timeout=10 [2026-05-08T08:47:09.670Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-05-08T08:47:09.670Z] > git --version # timeout=10 [2026-05-08T08:47:09.677Z] > git --version # 'git version 2.47.3' [2026-05-08T08:47:09.677Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-05-08T08:47:09.683Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-08T08:47:09.881Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-05-08T08:47:09.888Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-05-08T08:47:09.896Z] Avoid second fetch [2026-05-08T08:47:09.896Z] Checking out Revision ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f (main) [2026-05-08T08:47:09.896Z] > git config core.sparsecheckout # timeout=10 [2026-05-08T08:47:09.903Z] > git checkout -f ab1cefddbfd1e1b9bea9364b81ef15a6ee33407f # timeout=10 [2026-05-08T08:47:09.935Z] Commit message: "fn-testing: add allprio= to osp" [2026-05-08T08:47:09.997Z] [Pipeline] Start of Pipeline [2026-05-08T08:47:10.102Z] [Pipeline] node [2026-05-08T08:47:10.104Z] Running on anvil-ci-bm-phy04 in /srv/workspace/anvil/anvil-build-functional-testing [2026-05-08T08:47:10.105Z] [Pipeline] { [2026-05-08T08:47:10.113Z] [Pipeline] stage [2026-05-08T08:47:10.113Z] [Pipeline] { (Prep build env) [2026-05-08T08:47:10.122Z] [Pipeline] script [2026-05-08T08:47:10.124Z] [Pipeline] { [2026-05-08T08:47:10.129Z] [Pipeline] cleanWs [2026-05-08T08:47:10.179Z] [WS-CLEANUP] Deleting project workspace... [2026-05-08T08:47:10.179Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-05-08T08:47:10.203Z] [WS-CLEANUP] done [2026-05-08T08:47:10.220Z] [Pipeline] lock [2026-05-08T08:47:10.221Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-05-08T08:47:10.221Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-05-08T08:47:10.221Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-05-08T08:47:10.221Z] [Pipeline] { [2026-05-08T08:47:10.223Z] [Pipeline] } [2026-05-08T08:47:10.224Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-05-08T08:47:10.226Z] [Pipeline] // lock [2026-05-08T08:47:10.230Z] [Pipeline] } [2026-05-08T08:47:10.232Z] [Pipeline] // script [2026-05-08T08:47:10.234Z] [Pipeline] } [2026-05-08T08:47:10.235Z] [Pipeline] // stage [2026-05-08T08:47:10.238Z] [Pipeline] stage [2026-05-08T08:47:10.239Z] [Pipeline] { (Run functional testing) [2026-05-08T08:47:10.244Z] [Pipeline] script [2026-05-08T08:47:10.246Z] [Pipeline] { [2026-05-08T08:47:10.264Z] [Pipeline] echo [2026-05-08T08:47:10.264Z] runWithTimeout 360 [2026-05-08T08:47:10.265Z] [Pipeline] timeout [2026-05-08T08:47:10.265Z] Timeout set to expire in 6 hr 0 min [2026-05-08T08:47:10.265Z] [Pipeline] { [2026-05-08T08:47:10.270Z] [Pipeline] sh [2026-05-08T08:47:10.783Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 enterprise-staging none all [2026-05-08T08:47:10.783Z] CITBRANCH: main [2026-05-08T08:47:10.783Z] CITHOME: /root/ci-tools [2026-05-08T08:47:10.783Z] CITCMD: ci-setup-anvil-bm-vm [2026-05-08T08:47:10.783Z] CITCMDARGS: rhel-9 enterprise-staging none all [2026-05-08T08:47:10.783Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-08T08:47:10.783Z] Genering ssh keys and ssh config [2026-05-08T08:47:11.128Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:47:11.128Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:47:11.128Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:47:11.128Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:47:11.128Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:47:11.472Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-05-08T08:50:20.619Z] Destroying VMs (if any) [2026-05-08T08:50:20.619Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-08T08:50:20.619Z] Genering ssh keys and ssh config [2026-05-08T08:50:20.619Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:50:20.619Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:50:20.619Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:50:20.619Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:50:20.619Z] # github.com:22 SSH-2.0-00779af [2026-05-08T08:50:20.619Z] Cache already updated [2026-05-08T08:50:20.619Z] Creating nodes disks [2026-05-08T08:50:20.619Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-05-08T08:50:20.619Z] Generating an-test-deployX kickstart files/isos [2026-05-08T08:50:20.619Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-05-08T08:50:20.619Z] Media current: stdio file, overwriteable [2026-05-08T08:50:20.619Z] Media status : is blank [2026-05-08T08:50:20.619Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1720g free [2026-05-08T08:50:20.619Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] ISO image produced: 184 sectors [2026-05-08T08:50:20.619Z] Written to medium : 184 sectors at LBA 0 [2026-05-08T08:50:20.619Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-05-08T08:50:20.619Z] Media current: stdio file, overwriteable [2026-05-08T08:50:20.619Z] Media status : is blank [2026-05-08T08:50:20.619Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1729g free [2026-05-08T08:50:20.619Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] ISO image produced: 184 sectors [2026-05-08T08:50:20.619Z] Written to medium : 184 sectors at LBA 0 [2026-05-08T08:50:20.619Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-05-08T08:50:20.619Z] Media current: stdio file, overwriteable [2026-05-08T08:50:20.619Z] Media status : is blank [2026-05-08T08:50:20.619Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1729g free [2026-05-08T08:50:20.619Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] ISO image produced: 184 sectors [2026-05-08T08:50:20.619Z] Written to medium : 184 sectors at LBA 0 [2026-05-08T08:50:20.619Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-05-08T08:50:20.619Z] Media current: stdio file, overwriteable [2026-05-08T08:50:20.619Z] Media status : is blank [2026-05-08T08:50:20.619Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1729g free [2026-05-08T08:50:20.619Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] ISO image produced: 184 sectors [2026-05-08T08:50:20.619Z] Written to medium : 184 sectors at LBA 0 [2026-05-08T08:50:20.619Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-05-08T08:50:20.619Z] Media current: stdio file, overwriteable [2026-05-08T08:50:20.619Z] Media status : is blank [2026-05-08T08:50:20.619Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1729g free [2026-05-08T08:50:20.619Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-05-08T08:50:20.619Z] ISO image produced: 184 sectors [2026-05-08T08:50:20.619Z] Written to medium : 184 sectors at LBA 0 [2026-05-08T08:50:20.619Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-05-08T08:50:20.619Z] [2026-05-08T08:50:20.619Z] Generating cloud-init configs [2026-05-08T08:50:20.619Z] Generating cloud-init metadata and seed images [2026-05-08T08:50:20.619Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-08T08:50:20.619Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-08T08:50:20.619Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-08T08:50:20.619Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-08T08:50:20.619Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-05-08T08:50:20.619Z] Creating VMs [2026-05-08T08:50:21.813Z] [2026-05-08T08:50:21.813Z] Starting install... [2026-05-08T08:50:21.813Z] [2026-05-08T08:50:21.813Z] Creating domain... | 00:00 [2026-05-08T08:50:21.813Z] Domain creation completed. [2026-05-08T08:50:23.535Z] [2026-05-08T08:50:23.535Z] Starting install... [2026-05-08T08:50:23.535Z] [2026-05-08T08:50:23.535Z] Creating domain... | 00:00 [2026-05-08T08:50:23.535Z] Domain creation completed. [2026-05-08T08:50:25.257Z] [2026-05-08T08:50:25.257Z] Starting install... [2026-05-08T08:50:25.257Z] [2026-05-08T08:50:25.257Z] Creating domain... | 00:00 [2026-05-08T08:50:25.257Z] Domain creation completed. [2026-05-08T08:50:26.979Z] [2026-05-08T08:50:26.979Z] Starting install... [2026-05-08T08:50:26.979Z] [2026-05-08T08:50:26.979Z] Creating domain... | 00:00 [2026-05-08T08:50:26.979Z] Domain creation completed. [2026-05-08T08:50:28.704Z] [2026-05-08T08:50:28.704Z] Starting install... [2026-05-08T08:50:28.704Z] [2026-05-08T08:50:28.704Z] Creating domain... | 00:00 [2026-05-08T08:50:28.704Z] Domain creation completed. [2026-05-08T08:50:28.704Z] Waiting for nodes and cloud-init to complete full init [2026-05-08T08:50:31.757Z] cloud-init status node an-striker01: [2026-05-08T08:50:34.809Z] cloud-init status node an-striker02: [2026-05-08T08:50:37.862Z] cloud-init status node an-a01n01: [2026-05-08T08:50:41.753Z] cloud-init status node an-a01n02: [2026-05-08T08:50:44.089Z] cloud-init status node an-a01dr01: [2026-05-08T08:50:44.089Z] Fri May 8 04:50:44 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:50:57.541Z] cloud-init status node an-striker01: running [2026-05-08T08:50:57.541Z] cloud-init status node an-striker02: running [2026-05-08T08:50:57.890Z] cloud-init status node an-a01n01: running [2026-05-08T08:50:59.112Z] cloud-init status node an-a01n02: running [2026-05-08T08:50:59.872Z] cloud-init status node an-a01dr01: running [2026-05-08T08:50:59.872Z] Fri May 8 04:50:59 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:51:11.203Z] cloud-init status node an-striker01: running [2026-05-08T08:51:11.576Z] cloud-init status node an-striker02: running [2026-05-08T08:51:12.464Z] cloud-init status node an-a01n01: running [2026-05-08T08:51:13.241Z] cloud-init status node an-a01n02: running [2026-05-08T08:51:14.517Z] cloud-init status node an-a01dr01: running [2026-05-08T08:51:14.517Z] Fri May 8 04:51:14 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:51:26.232Z] cloud-init status node an-striker01: running [2026-05-08T08:51:26.232Z] cloud-init status node an-striker02: running [2026-05-08T08:51:27.014Z] cloud-init status node an-a01n01: running [2026-05-08T08:51:27.917Z] cloud-init status node an-a01n02: running [2026-05-08T08:51:28.704Z] cloud-init status node an-a01dr01: running [2026-05-08T08:51:28.704Z] Fri May 8 04:51:28 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:51:40.073Z] cloud-init status node an-striker01: running [2026-05-08T08:51:40.840Z] cloud-init status node an-striker02: running [2026-05-08T08:51:42.080Z] cloud-init status node an-a01n01: running [2026-05-08T08:51:42.847Z] cloud-init status node an-a01n02: running [2026-05-08T08:51:43.965Z] cloud-init status node an-a01dr01: running [2026-05-08T08:51:43.965Z] Fri May 8 04:51:43 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:52:01.899Z] cloud-init status node an-striker01: running [2026-05-08T08:52:01.899Z] cloud-init status node an-striker02: running [2026-05-08T08:52:01.899Z] cloud-init status node an-a01n01: running [2026-05-08T08:52:02.696Z] cloud-init status node an-a01n02: running [2026-05-08T08:52:04.431Z] cloud-init status node an-a01dr01: running [2026-05-08T08:52:04.431Z] Fri May 8 04:52:03 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:52:14.549Z] cloud-init status node an-striker01: running [2026-05-08T08:52:16.053Z] cloud-init status node an-striker02: running [2026-05-08T08:52:21.273Z] cloud-init status node an-a01n01: running [2026-05-08T08:52:28.470Z] cloud-init status node an-a01n02: running [2026-05-08T08:52:28.470Z] cloud-init status node an-a01dr01: running [2026-05-08T08:52:28.470Z] Fri May 8 04:52:27 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:52:40.725Z] cloud-init status node an-striker01: running [2026-05-08T08:52:40.725Z] cloud-init status node an-striker02: running [2026-05-08T08:52:40.725Z] cloud-init status node an-a01n01: running [2026-05-08T08:52:42.067Z] cloud-init status node an-a01n02: running [2026-05-08T08:52:43.289Z] cloud-init status node an-a01dr01: running [2026-05-08T08:52:43.289Z] Fri May 8 04:52:42 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:52:54.227Z] cloud-init status node an-striker01: running [2026-05-08T08:52:55.109Z] cloud-init status node an-striker02: running [2026-05-08T08:52:55.608Z] cloud-init status node an-a01n01: running [2026-05-08T08:52:56.625Z] cloud-init status node an-a01n02: running [2026-05-08T08:52:57.696Z] cloud-init status node an-a01dr01: running [2026-05-08T08:52:57.696Z] Fri May 8 04:52:57 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:53:09.273Z] cloud-init status node an-striker01: running [2026-05-08T08:53:09.640Z] cloud-init status node an-striker02: running [2026-05-08T08:53:11.209Z] cloud-init status node an-a01n01: running [2026-05-08T08:53:11.209Z] cloud-init status node an-a01n02: running [2026-05-08T08:53:12.171Z] cloud-init status node an-a01dr01: running [2026-05-08T08:53:12.171Z] Fri May 8 04:53:11 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:53:23.067Z] cloud-init status node an-striker01: running [2026-05-08T08:53:24.089Z] cloud-init status node an-striker02: running [2026-05-08T08:53:24.937Z] cloud-init status node an-a01n01: running [2026-05-08T08:53:27.128Z] cloud-init status node an-a01n02: running [2026-05-08T08:53:27.128Z] cloud-init status node an-a01dr01: running [2026-05-08T08:53:27.128Z] Fri May 8 04:53:26 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:53:43.499Z] cloud-init status node an-striker01: running [2026-05-08T08:53:44.265Z] cloud-init status node an-striker02: running [2026-05-08T08:53:45.077Z] cloud-init status node an-a01n01: running [2026-05-08T08:53:45.659Z] cloud-init status node an-a01n02: running [2026-05-08T08:53:56.973Z] cloud-init status node an-a01dr01: [2026-05-08T08:53:56.973Z] Fri May 8 04:53:55 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:54:08.178Z] cloud-init status node an-striker01: running [2026-05-08T08:54:08.178Z] cloud-init status node an-striker02: running [2026-05-08T08:54:08.178Z] cloud-init status node an-a01n01: running [2026-05-08T08:54:09.351Z] cloud-init status node an-a01n02: running [2026-05-08T08:54:10.112Z] cloud-init status node an-a01dr01: running [2026-05-08T08:54:10.112Z] Fri May 8 04:54:09 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:54:20.957Z] cloud-init status node an-striker01: running [2026-05-08T08:54:22.713Z] cloud-init status node an-striker02: running [2026-05-08T08:54:23.543Z] cloud-init status node an-a01n01: running [2026-05-08T08:54:24.412Z] cloud-init status node an-a01n02: running [2026-05-08T08:54:25.127Z] cloud-init status node an-a01dr01: running [2026-05-08T08:54:25.127Z] Fri May 8 04:54:24 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:54:35.943Z] cloud-init status node an-striker01: running [2026-05-08T08:54:36.915Z] cloud-init status node an-striker02: running [2026-05-08T08:54:38.742Z] cloud-init status node an-a01n01: running [2026-05-08T08:54:38.742Z] cloud-init status node an-a01n02: running [2026-05-08T08:54:40.096Z] cloud-init status node an-a01dr01: running [2026-05-08T08:54:40.096Z] Fri May 8 04:54:39 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:54:50.692Z] cloud-init status node an-striker01: running [2026-05-08T08:54:51.533Z] cloud-init status node an-striker02: running [2026-05-08T08:54:52.701Z] cloud-init status node an-a01n01: running [2026-05-08T08:54:53.675Z] cloud-init status node an-a01n02: running [2026-05-08T08:54:54.858Z] cloud-init status node an-a01dr01: running [2026-05-08T08:54:54.858Z] Fri May 8 04:54:54 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:55:07.062Z] cloud-init status node an-striker01: running [2026-05-08T08:55:07.062Z] cloud-init status node an-striker02: running [2026-05-08T08:55:07.062Z] cloud-init status node an-a01n01: running [2026-05-08T08:55:07.905Z] cloud-init status node an-a01n02: running [2026-05-08T08:55:08.895Z] cloud-init status node an-a01dr01: running [2026-05-08T08:55:08.895Z] Fri May 8 04:55:08 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:55:21.116Z] cloud-init status node an-striker01: running [2026-05-08T08:55:21.116Z] cloud-init status node an-striker02: running [2026-05-08T08:55:21.905Z] cloud-init status node an-a01n01: running [2026-05-08T08:55:22.673Z] cloud-init status node an-a01n02: running [2026-05-08T08:55:23.446Z] cloud-init status node an-a01dr01: running [2026-05-08T08:55:23.446Z] Fri May 8 04:55:23 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:55:34.716Z] cloud-init status node an-striker01: running [2026-05-08T08:55:35.483Z] cloud-init status node an-striker02: running [2026-05-08T08:55:36.255Z] cloud-init status node an-a01n01: running [2026-05-08T08:55:37.107Z] cloud-init status node an-a01n02: running [2026-05-08T08:55:38.172Z] cloud-init status node an-a01dr01: running [2026-05-08T08:55:38.172Z] Fri May 8 04:55:37 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:55:49.422Z] cloud-init status node an-striker01: running [2026-05-08T08:55:49.784Z] cloud-init status node an-striker02: running [2026-05-08T08:55:50.550Z] cloud-init status node an-a01n01: running [2026-05-08T08:55:51.788Z] cloud-init status node an-a01n02: running [2026-05-08T08:55:52.553Z] cloud-init status node an-a01dr01: running [2026-05-08T08:55:52.553Z] Fri May 8 04:55:52 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:56:03.766Z] cloud-init status node an-striker01: running [2026-05-08T08:56:04.125Z] cloud-init status node an-striker02: running [2026-05-08T08:56:05.263Z] cloud-init status node an-a01n01: running [2026-05-08T08:56:06.029Z] cloud-init status node an-a01n02: running [2026-05-08T08:56:07.274Z] cloud-init status node an-a01dr01: running [2026-05-08T08:56:07.274Z] Fri May 8 04:56:06 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:56:18.560Z] cloud-init status node an-striker01: running [2026-05-08T08:56:18.918Z] cloud-init status node an-striker02: running [2026-05-08T08:56:19.672Z] cloud-init status node an-a01n01: running [2026-05-08T08:56:20.912Z] cloud-init status node an-a01n02: running [2026-05-08T08:56:21.675Z] cloud-init status node an-a01dr01: running [2026-05-08T08:56:21.675Z] Fri May 8 04:56:21 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:56:33.053Z] cloud-init status node an-striker01: running [2026-05-08T08:56:33.408Z] cloud-init status node an-striker02: running [2026-05-08T08:56:34.706Z] cloud-init status node an-a01n01: running [2026-05-08T08:56:35.475Z] cloud-init status node an-a01n02: running [2026-05-08T08:56:36.224Z] cloud-init status node an-a01dr01: running [2026-05-08T08:56:36.224Z] Fri May 8 04:56:36 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:56:47.260Z] cloud-init status node an-striker01: running [2026-05-08T08:56:48.458Z] cloud-init status node an-striker02: running [2026-05-08T08:56:49.200Z] cloud-init status node an-a01n01: running [2026-05-08T08:56:50.398Z] cloud-init status node an-a01n02: running [2026-05-08T08:56:51.141Z] cloud-init status node an-a01dr01: running [2026-05-08T08:56:51.141Z] Fri May 8 04:56:50 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:57:02.326Z] cloud-init status node an-striker01: running [2026-05-08T08:57:03.066Z] cloud-init status node an-striker02: running [2026-05-08T08:57:04.262Z] cloud-init status node an-a01n01: running [2026-05-08T08:57:05.003Z] cloud-init status node an-a01n02: running [2026-05-08T08:57:05.745Z] cloud-init status node an-a01dr01: running [2026-05-08T08:57:05.745Z] Fri May 8 04:57:05 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:57:16.785Z] cloud-init status node an-striker01: running [2026-05-08T08:57:17.981Z] cloud-init status node an-striker02: running [2026-05-08T08:57:19.181Z] cloud-init status node an-a01n01: running [2026-05-08T08:57:19.922Z] cloud-init status node an-a01n02: running [2026-05-08T08:57:20.662Z] cloud-init status node an-a01dr01: running [2026-05-08T08:57:20.662Z] Fri May 8 04:57:20 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:57:31.679Z] cloud-init status node an-striker01: running [2026-05-08T08:57:32.420Z] cloud-init status node an-striker02: running [2026-05-08T08:57:33.615Z] cloud-init status node an-a01n01: running [2026-05-08T08:57:34.356Z] cloud-init status node an-a01n02: running [2026-05-08T08:57:35.553Z] cloud-init status node an-a01dr01: running [2026-05-08T08:57:35.553Z] Fri May 8 04:57:35 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:57:46.710Z] cloud-init status node an-striker01: running [2026-05-08T08:57:47.056Z] cloud-init status node an-striker02: running [2026-05-08T08:57:48.252Z] cloud-init status node an-a01n01: running [2026-05-08T08:57:48.994Z] cloud-init status node an-a01n02: running [2026-05-08T08:57:49.735Z] cloud-init status node an-a01dr01: running [2026-05-08T08:57:49.735Z] Fri May 8 04:57:49 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:58:00.762Z] cloud-init status node an-striker01: running [2026-05-08T08:58:01.503Z] cloud-init status node an-striker02: running [2026-05-08T08:58:02.700Z] cloud-init status node an-a01n01: running [2026-05-08T08:58:03.439Z] cloud-init status node an-a01n02: running [2026-05-08T08:58:04.634Z] cloud-init status node an-a01dr01: running [2026-05-08T08:58:04.634Z] Fri May 8 04:58:04 EDT 2026 actual: 0 expected: 5 [2026-05-08T08:58:15.789Z] cloud-init status node an-striker01: running [2026-05-08T08:58:16.534Z] cloud-init status node an-striker02: running [2026-05-08T08:58:17.732Z] cloud-init status node an-a01n01: running [2026-05-08T08:58:18.474Z] cloud-init status node an-a01n02: running [2026-05-08T08:58:19.672Z] cloud-init status node an-a01dr01: done [2026-05-08T08:58:19.672Z] Fri May 8 04:58:19 EDT 2026 actual: 1 expected: 5 [2026-05-08T08:58:30.692Z] cloud-init status node an-striker01: running [2026-05-08T08:58:32.028Z] cloud-init status node an-striker02: running [2026-05-08T08:58:33.756Z] cloud-init status node an-a01n01: running [2026-05-08T08:58:34.498Z] cloud-init status node an-a01n02: done [2026-05-08T08:58:35.696Z] cloud-init status node an-a01dr01: done [2026-05-08T08:58:35.696Z] Fri May 8 04:58:35 EDT 2026 actual: 2 expected: 5 [2026-05-08T08:58:49.039Z] cloud-init status node an-striker01: running [2026-05-08T08:58:49.039Z] cloud-init status node an-striker02: running [2026-05-08T08:58:49.385Z] cloud-init status node an-a01n01: done [2026-05-08T08:58:50.584Z] cloud-init status node an-a01n02: done [2026-05-08T08:58:51.781Z] cloud-init status node an-a01dr01: done [2026-05-08T08:58:51.781Z] Fri May 8 04:58:51 EDT 2026 actual: 3 expected: 5 [2026-05-08T08:59:02.939Z] cloud-init status node an-striker01: running [2026-05-08T08:59:03.681Z] cloud-init status node an-striker02: running [2026-05-08T08:59:04.880Z] cloud-init status node an-a01n01: done [2026-05-08T08:59:05.621Z] cloud-init status node an-a01n02: done [2026-05-08T08:59:06.819Z] cloud-init status node an-a01dr01: done [2026-05-08T08:59:06.819Z] Fri May 8 04:59:06 EDT 2026 actual: 3 expected: 5 [2026-05-08T08:59:17.842Z] cloud-init status node an-striker01: running [2026-05-08T08:59:18.585Z] cloud-init status node an-striker02: running [2026-05-08T08:59:19.783Z] cloud-init status node an-a01n01: done [2026-05-08T08:59:20.526Z] cloud-init status node an-a01n02: done [2026-05-08T08:59:21.272Z] cloud-init status node an-a01dr01: done [2026-05-08T08:59:21.272Z] Fri May 8 04:59:21 EDT 2026 actual: 3 expected: 5 [2026-05-08T08:59:34.611Z] cloud-init status node an-striker01: done [2026-05-08T08:59:34.611Z] cloud-init status node an-striker02: running [2026-05-08T08:59:34.956Z] cloud-init status node an-a01n01: done [2026-05-08T08:59:35.698Z] cloud-init status node an-a01n02: done [2026-05-08T08:59:36.895Z] cloud-init status node an-a01dr01: done [2026-05-08T08:59:36.895Z] Fri May 8 04:59:36 EDT 2026 actual: 4 expected: 5 [2026-05-08T08:59:47.921Z] cloud-init status node an-striker01: done [2026-05-08T08:59:48.664Z] cloud-init status node an-striker02: done [2026-05-08T08:59:49.525Z] cloud-init status node an-a01n01: done [2026-05-08T08:59:50.723Z] cloud-init status node an-a01n02: done [2026-05-08T08:59:51.464Z] cloud-init status node an-a01dr01: done [2026-05-08T08:59:51.464Z] Fri May 8 04:59:51 EDT 2026 actual: 5 expected: 5 [2026-05-08T08:59:51.464Z] Building simengine model [2026-05-08T09:00:04.799Z] Generating anvil striker auto config [2026-05-08T09:00:08.698Z] Cloning into 'anvil-test-suite'... [2026-05-08T09:00:08.698Z] generated ansible config: [2026-05-08T09:00:08.698Z] ========================= [2026-05-08T09:00:08.698Z] [defaults] [2026-05-08T09:00:08.698Z] inventory = ansible-inventory [2026-05-08T09:00:08.698Z] forks = 500 [2026-05-08T09:00:08.698Z] any_errors_fatal = true [2026-05-08T09:00:08.698Z] display_args_to_stdout = true [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:08.698Z] [ssh_connection] [2026-05-08T09:00:08.698Z] retries = 20 [2026-05-08T09:00:08.698Z] ========================= [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:08.698Z] generated ansible inventory: [2026-05-08T09:00:08.698Z] ============================ [2026-05-08T09:00:08.698Z] [strikers] [2026-05-08T09:00:08.698Z] an-striker01 [2026-05-08T09:00:08.698Z] an-striker02 [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:08.698Z] [nodes] [2026-05-08T09:00:08.698Z] an-a01n01 [2026-05-08T09:00:08.698Z] an-a01n02 [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:08.698Z] [drnodes] [2026-05-08T09:00:08.698Z] ============================ [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:08.698Z] Checking connectivity to nodes [2026-05-08T09:00:08.698Z] [2026-05-08T09:00:11.036Z] an-a01n02 | SUCCESS => { [2026-05-08T09:00:11.036Z] "ansible_facts": { [2026-05-08T09:00:11.036Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-08T09:00:11.036Z] }, [2026-05-08T09:00:11.036Z] "changed": false, [2026-05-08T09:00:11.036Z] "ping": "pong" [2026-05-08T09:00:11.036Z] } [2026-05-08T09:00:11.036Z] an-a01n01 | SUCCESS => { [2026-05-08T09:00:11.036Z] "ansible_facts": { [2026-05-08T09:00:11.036Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-08T09:00:11.036Z] }, [2026-05-08T09:00:11.036Z] "changed": false, [2026-05-08T09:00:11.036Z] "ping": "pong" [2026-05-08T09:00:11.036Z] } [2026-05-08T09:00:11.036Z] an-striker01 | SUCCESS => { [2026-05-08T09:00:11.036Z] "ansible_facts": { [2026-05-08T09:00:11.036Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-08T09:00:11.036Z] }, [2026-05-08T09:00:11.036Z] "changed": false, [2026-05-08T09:00:11.036Z] "ping": "pong" [2026-05-08T09:00:11.036Z] } [2026-05-08T09:00:11.036Z] an-striker02 | SUCCESS => { [2026-05-08T09:00:11.036Z] "ansible_facts": { [2026-05-08T09:00:11.036Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-05-08T09:00:11.036Z] }, [2026-05-08T09:00:11.036Z] "changed": false, [2026-05-08T09:00:11.036Z] "ping": "pong" [2026-05-08T09:00:11.036Z] } [2026-05-08T09:00:11.036Z] [2026-05-08T09:00:11.776Z] [2026-05-08T09:00:11.776Z] PLAY [Setup all nodes] ********************************************************* [2026-05-08T09:00:11.776Z] [2026-05-08T09:00:11.776Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:00:14.232Z] ok: [an-striker01] [2026-05-08T09:00:14.232Z] ok: [an-striker02] [2026-05-08T09:00:14.232Z] ok: [an-a01n02] [2026-05-08T09:00:14.232Z] ok: [an-a01n01] [2026-05-08T09:00:14.232Z] [2026-05-08T09:00:14.232Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-05-08T09:00:15.430Z] changed: [an-a01n02] [2026-05-08T09:00:15.430Z] changed: [an-a01n01] [2026-05-08T09:00:15.430Z] changed: [an-striker02] [2026-05-08T09:00:15.430Z] changed: [an-striker01] [2026-05-08T09:00:15.430Z] [2026-05-08T09:00:15.430Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-05-08T09:00:16.170Z] ok: [an-a01n02] [2026-05-08T09:00:16.170Z] ok: [an-a01n01] [2026-05-08T09:00:16.170Z] ok: [an-striker01] [2026-05-08T09:00:16.170Z] ok: [an-striker02] [2026-05-08T09:00:16.170Z] [2026-05-08T09:00:16.170Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-05-08T09:00:17.896Z] changed: [an-a01n01] [2026-05-08T09:00:17.896Z] changed: [an-a01n02] [2026-05-08T09:00:17.896Z] changed: [an-striker02] [2026-05-08T09:00:17.896Z] changed: [an-striker01] [2026-05-08T09:00:17.896Z] [2026-05-08T09:00:17.896Z] PLAY [setup strikers] ********************************************************** [2026-05-08T09:00:17.896Z] [2026-05-08T09:00:17.896Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:00:19.094Z] ok: [an-striker01] [2026-05-08T09:00:19.094Z] ok: [an-striker02] [2026-05-08T09:00:19.094Z] [2026-05-08T09:00:19.094Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-05-08T09:00:20.820Z] changed: [an-striker01] [2026-05-08T09:00:20.820Z] changed: [an-striker02] [2026-05-08T09:00:20.820Z] [2026-05-08T09:00:20.820Z] PLAY [setup lvm storage] ******************************************************* [2026-05-08T09:00:20.820Z] [2026-05-08T09:00:20.820Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:00:22.547Z] ok: [an-a01n01] [2026-05-08T09:00:22.547Z] ok: [an-a01n02] [2026-05-08T09:00:22.547Z] [2026-05-08T09:00:22.547Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-05-08T09:00:22.548Z] ] **************** [2026-05-08T09:00:23.291Z] ok: [an-a01n02] [2026-05-08T09:00:23.291Z] ok: [an-a01n01] [2026-05-08T09:00:23.291Z] [2026-05-08T09:00:23.291Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-05-08T09:00:23.291Z] skipping: [an-a01n01] [2026-05-08T09:00:23.291Z] [2026-05-08T09:00:23.291Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-05-08T09:00:24.490Z] changed: [an-a01n01] [2026-05-08T09:00:24.490Z] changed: [an-a01n02] [2026-05-08T09:00:24.490Z] [2026-05-08T09:00:24.490Z] PLAY [deploy anvil] ************************************************************ [2026-05-08T09:00:24.490Z] [2026-05-08T09:00:24.490Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:00:26.214Z] ok: [an-striker02] [2026-05-08T09:00:26.214Z] ok: [an-striker01] [2026-05-08T09:00:26.214Z] [2026-05-08T09:00:26.214Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-05-08T09:00:26.956Z] ok: [an-striker02] [2026-05-08T09:00:26.956Z] ok: [an-striker01] [2026-05-08T09:00:26.956Z] [2026-05-08T09:00:26.956Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-05-08T09:00:28.698Z] changed: [an-striker02] [2026-05-08T09:00:28.698Z] changed: [an-striker01] [2026-05-08T09:00:28.698Z] [2026-05-08T09:00:28.698Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-05-08T09:00:28.698Z] ] *** [2026-05-08T09:00:31.750Z] changed: [an-striker02] [2026-05-08T09:00:31.750Z] changed: [an-striker01] [2026-05-08T09:00:31.750Z] [2026-05-08T09:00:31.750Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:00:31.750Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:00:31.750Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-05-08T09:00:31.750Z] ] *** [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-05-08T09:05:56.860Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-05-08T09:05:56.860Z] ok: [an-striker02 -> localhost] [2026-05-08T09:05:56.860Z] ok: [an-striker01 -> localhost] [2026-05-08T09:05:56.860Z] [2026-05-08T09:05:56.860Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-05-08T09:05:56.860Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-05-08T09:05:56.860Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-05-08T09:05:56.860Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-05-08T09:05:56.860Z] echo "DB has completed sync" [2026-05-08T09:05:56.860Z] exit 0 [2026-05-08T09:05:56.860Z] fi [2026-05-08T09:05:56.860Z] if [ "$dbcon" != 2 ]; then [2026-05-08T09:05:56.860Z] echo "Detected $dbcon db connections (expected 2)" [2026-05-08T09:05:56.860Z] exit 1 [2026-05-08T09:05:56.860Z] fi [2026-05-08T09:05:56.860Z] ] *** [2026-05-08T09:06:30.506Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-05-08T09:06:30.506Z] ok: [an-striker01] [2026-05-08T09:06:30.506Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-05-08T09:06:30.506Z] ok: [an-striker02] [2026-05-08T09:06:30.506Z] [2026-05-08T09:06:30.506Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:06:30.506Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:06:30.506Z] --username=postgres anvil -t \ [2026-05-08T09:06:30.506Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-05-08T09:06:30.506Z] grep altee | \ [2026-05-08T09:06:30.506Z] grep 1$ | \ [2026-05-08T09:06:30.506Z] wc -l [2026-05-08T09:06:30.506Z] ] *** [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-05-08T09:12:10.614Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-05-08T09:12:10.615Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-05-08T09:16:05.030Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-05-08T09:16:05.031Z] ok: [an-striker02 -> localhost] [2026-05-08T09:16:05.031Z] ok: [an-striker01 -> localhost] [2026-05-08T09:16:05.031Z] [2026-05-08T09:16:05.031Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:16:05.031Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:16:05.031Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-05-08T09:16:05.031Z] grep altee | \ [2026-05-08T09:16:05.031Z] wc -l [2026-05-08T09:16:05.031Z] ] *** [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-05-08T09:16:05.031Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for anvil nodes to form (120 retries left). [2026-05-08T09:16:05.031Z] ok: [an-striker01 -> localhost] [2026-05-08T09:16:05.031Z] ok: [an-striker02 -> localhost] [2026-05-08T09:16:05.031Z] [2026-05-08T09:16:05.031Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:16:05.031Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:16:05.031Z] --username=postgres anvil -t \ [2026-05-08T09:16:05.031Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-05-08T09:16:05.031Z] grep altee | \ [2026-05-08T09:16:05.031Z] grep 1$ | \ [2026-05-08T09:16:05.031Z] wc -l [2026-05-08T09:16:05.031Z] ] *** [2026-05-08T09:17:58.782Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-05-08T09:17:58.782Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-05-08T09:17:58.782Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-05-08T09:17:58.783Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-05-08T09:17:58.783Z] ok: [an-striker01 -> localhost] [2026-05-08T09:17:58.783Z] ok: [an-striker02 -> localhost] [2026-05-08T09:17:58.783Z] [2026-05-08T09:17:58.783Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:17:58.783Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:17:58.783Z] --username=postgres anvil -t \ [2026-05-08T09:17:58.783Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-05-08T09:17:58.783Z] grep altee | \ [2026-05-08T09:17:58.783Z] grep 1$ | \ [2026-05-08T09:17:58.783Z] wc -l [2026-05-08T09:17:58.783Z] ] *** [2026-05-08T09:18:06.212Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-05-08T09:18:06.212Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-05-08T09:18:06.212Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-05-08T09:18:06.212Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-05-08T09:18:06.212Z] ok: [an-striker01 -> localhost] [2026-05-08T09:18:06.212Z] ok: [an-striker02 -> localhost] [2026-05-08T09:18:06.212Z] [2026-05-08T09:18:06.212Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-05-08T09:18:06.212Z] psql --host="{{ inventory_hostname }}" \ [2026-05-08T09:18:06.212Z] --username=postgres anvil -t \ [2026-05-08T09:18:06.212Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-05-08T09:18:06.212Z] grep altee | \ [2026-05-08T09:18:06.212Z] grep 1$ | \ [2026-05-08T09:18:06.212Z] wc -l [2026-05-08T09:18:06.212Z] ] *** [2026-05-08T09:18:06.652Z] ok: [an-striker01 -> localhost] [2026-05-08T09:18:06.652Z] ok: [an-striker02 -> localhost] [2026-05-08T09:18:06.652Z] [2026-05-08T09:18:06.652Z] PLAY [check network status after deployment] *********************************** [2026-05-08T09:18:06.652Z] [2026-05-08T09:18:06.652Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:18:08.988Z] ok: [an-a01n02] [2026-05-08T09:18:08.988Z] ok: [an-a01n01] [2026-05-08T09:18:08.988Z] ok: [an-striker02] [2026-05-08T09:18:08.988Z] ok: [an-striker01] [2026-05-08T09:18:08.988Z] [2026-05-08T09:18:08.988Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-05-08T09:18:08.988Z] if [ "$device" = "" ]; then [2026-05-08T09:18:08.988Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-05-08T09:18:08.988Z] exit 1 [2026-05-08T09:18:08.988Z] fi [2026-05-08T09:18:08.988Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-05-08T09:18:08.988Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-05-08T09:18:08.988Z] exit 1 [2026-05-08T09:18:08.988Z] fi [2026-05-08T09:18:08.988Z] exit 0 [2026-05-08T09:18:08.988Z] ] *** [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=bcn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-05-08T09:18:16.415Z] skipping: [an-a01n01] => (item=lo) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=ifn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=sn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=bcn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=ifn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=bcn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=sn1_link2) [2026-05-08T09:18:16.415Z] skipping: [an-a01n02] => (item=lo) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=bcn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=sn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=sn1_link1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=ifn1_link2) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-striker02] => (item=ifn1_link2) [2026-05-08T09:18:16.415Z] skipping: [an-striker02] => (item=lo) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-05-08T09:18:16.415Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-05-08T09:18:16.415Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-05-08T09:18:16.415Z] skipping: [an-striker01] => (item=lo) [2026-05-08T09:18:16.415Z] [2026-05-08T09:18:16.415Z] PLAY [check pacemaker status after deployment] ********************************* [2026-05-08T09:18:16.415Z] [2026-05-08T09:18:16.415Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:18:18.136Z] ok: [an-a01n02] [2026-05-08T09:18:18.136Z] ok: [an-a01n01] [2026-05-08T09:18:18.136Z] [2026-05-08T09:18:18.136Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-05-08T09:18:18.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T09:18:18.136Z] [2026-05-08T09:18:18.136Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T09:18:18.136Z] ] *** [2026-05-08T09:18:22.027Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T09:18:22.027Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T09:18:22.027Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T09:18:22.027Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T09:18:22.027Z] [2026-05-08T09:18:22.027Z] PLAY [check drbd kmod] ********************************************************* [2026-05-08T09:18:22.027Z] [2026-05-08T09:18:22.027Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:18:23.221Z] ok: [an-a01n02] [2026-05-08T09:18:23.221Z] ok: [an-a01n01] [2026-05-08T09:18:23.221Z] [2026-05-08T09:18:23.221Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-05-08T09:18:23.221Z] ] *** [2026-05-08T09:18:23.961Z] ok: [an-a01n01] [2026-05-08T09:18:23.961Z] ok: [an-a01n02] [2026-05-08T09:18:23.961Z] [2026-05-08T09:18:23.961Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-05-08T09:18:23.961Z] ] *********************** [2026-05-08T09:18:23.961Z] skipping: [an-a01n01] [2026-05-08T09:18:23.961Z] skipping: [an-a01n02] [2026-05-08T09:18:23.961Z] [2026-05-08T09:18:23.961Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-05-08T09:18:23.961Z] ] *** [2026-05-08T09:18:23.961Z] skipping: [an-a01n01] [2026-05-08T09:18:23.961Z] skipping: [an-a01n02] [2026-05-08T09:18:23.961Z] [2026-05-08T09:18:23.961Z] PLAY [Copy iso images to striker01] ******************************************** [2026-05-08T09:18:23.961Z] [2026-05-08T09:18:23.961Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:18:25.776Z] ok: [an-striker01] [2026-05-08T09:18:25.776Z] [2026-05-08T09:18:25.776Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-05-08T09:18:26.120Z] ok: [an-striker01] [2026-05-08T09:18:26.120Z] [2026-05-08T09:18:26.120Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-05-08T09:18:27.315Z] changed: [an-striker01] [2026-05-08T09:18:27.315Z] [2026-05-08T09:18:27.315Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-05-08T09:18:36.398Z] changed: [an-striker01] => (item=deploy1.iso) [2026-05-08T09:18:36.398Z] changed: [an-striker01] => (item=deploy2.iso) [2026-05-08T09:18:36.398Z] changed: [an-striker01] => (item=deploy3.iso) [2026-05-08T09:18:36.398Z] changed: [an-striker01] => (item=deploy4.iso) [2026-05-08T09:18:36.398Z] changed: [an-striker01] => (item=deploy5.iso) [2026-05-08T09:18:36.398Z] [2026-05-08T09:18:36.398Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-05-08T09:18:36.398Z] ] *** [2026-05-08T09:18:36.398Z] changed: [an-striker01] [2026-05-08T09:18:36.398Z] [2026-05-08T09:18:36.398Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-05-08T09:18:36.398Z] changed: [an-striker01] [2026-05-08T09:18:36.398Z] [2026-05-08T09:18:36.398Z] PLAY [Check iso images propagation] ******************************************** [2026-05-08T09:18:36.398Z] [2026-05-08T09:18:36.398Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:18:38.733Z] ok: [an-a01n02] [2026-05-08T09:18:38.733Z] ok: [an-a01n01] [2026-05-08T09:18:38.733Z] ok: [an-striker02] [2026-05-08T09:18:38.733Z] ok: [an-striker01] [2026-05-08T09:18:38.733Z] [2026-05-08T09:18:38.733Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-05-08T09:18:38.733Z] exit 1 [2026-05-08T09:18:38.733Z] else [2026-05-08T09:18:38.733Z] exit 0 [2026-05-08T09:18:38.733Z] fi [2026-05-08T09:18:38.733Z] ] *** [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-08T09:20:17.251Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-05-08T09:20:17.251Z] ok: [an-striker01] [2026-05-08T09:20:17.251Z] ok: [an-a01n01] [2026-05-08T09:20:17.251Z] ok: [an-a01n02] [2026-05-08T09:20:17.251Z] ok: [an-striker02] [2026-05-08T09:20:17.251Z] [2026-05-08T09:20:17.251Z] PLAY [example tests] *********************************************************** [2026-05-08T09:20:17.251Z] [2026-05-08T09:20:17.251Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:20:18.973Z] ok: [an-striker01] [2026-05-08T09:20:18.973Z] ok: [an-striker02] [2026-05-08T09:20:18.973Z] ok: [an-a01n02] [2026-05-08T09:20:18.973Z] ok: [an-a01n01] [2026-05-08T09:20:18.973Z] [2026-05-08T09:20:18.973Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-05-08T09:20:18.973Z] ] *** [2026-05-08T09:20:19.713Z] skipping: [an-a01n01] [2026-05-08T09:20:19.713Z] skipping: [an-a01n02] [2026-05-08T09:20:19.713Z] ok: [an-striker01] [2026-05-08T09:20:19.713Z] ok: [an-striker02] [2026-05-08T09:20:19.713Z] [2026-05-08T09:20:19.713Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-05-08T09:20:19.713Z] ] *** [2026-05-08T09:20:19.713Z] skipping: [an-striker01] [2026-05-08T09:20:19.713Z] skipping: [an-striker02] [2026-05-08T09:20:20.453Z] ok: [an-a01n01] [2026-05-08T09:20:20.453Z] ok: [an-a01n02] [2026-05-08T09:20:20.453Z] [2026-05-08T09:20:20.453Z] TASK [inject failure _raw_params=/usr/bin/false [2026-05-08T09:20:20.453Z] ] ******************************* [2026-05-08T09:20:20.453Z] skipping: [an-striker01] [2026-05-08T09:20:20.453Z] skipping: [an-striker02] [2026-05-08T09:20:20.797Z] skipping: [an-a01n01] [2026-05-08T09:20:20.797Z] skipping: [an-a01n02] [2026-05-08T09:20:20.797Z] [2026-05-08T09:20:20.797Z] PLAY [Record anvil available resources from striker01] ************************* [2026-05-08T09:20:20.797Z] [2026-05-08T09:20:20.797Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:20:22.518Z] ok: [an-striker01] [2026-05-08T09:20:22.518Z] [2026-05-08T09:20:22.518Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-05-08T09:20:22.518Z] ] *** [2026-05-08T09:20:25.567Z] ok: [an-striker01] [2026-05-08T09:20:25.567Z] [2026-05-08T09:20:25.567Z] PLAY [check selinux status] **************************************************** [2026-05-08T09:20:25.567Z] [2026-05-08T09:20:25.567Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:20:27.902Z] ok: [an-striker02] [2026-05-08T09:20:27.903Z] ok: [an-a01n02] [2026-05-08T09:20:27.903Z] ok: [an-a01n01] [2026-05-08T09:20:27.903Z] ok: [an-striker01] [2026-05-08T09:20:27.903Z] [2026-05-08T09:20:27.903Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-05-08T09:20:27.903Z] skipping: [an-striker01] [2026-05-08T09:20:27.903Z] skipping: [an-striker02] [2026-05-08T09:20:27.903Z] skipping: [an-a01n01] [2026-05-08T09:20:27.903Z] skipping: [an-a01n02] [2026-05-08T09:20:27.903Z] [2026-05-08T09:20:27.903Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-05-08T09:20:27.903Z] ok: [an-striker01] => { [2026-05-08T09:20:27.903Z] "ansible_selinux": { [2026-05-08T09:20:27.903Z] "config_mode": "enforcing", [2026-05-08T09:20:27.903Z] "mode": "enforcing", [2026-05-08T09:20:27.903Z] "policyvers": 33, [2026-05-08T09:20:27.903Z] "status": "enabled", [2026-05-08T09:20:27.903Z] "type": "targeted" [2026-05-08T09:20:27.903Z] }, [2026-05-08T09:20:27.903Z] "failed_when_result": false [2026-05-08T09:20:27.903Z] } [2026-05-08T09:20:27.903Z] ok: [an-striker02] => { [2026-05-08T09:20:27.903Z] "ansible_selinux": { [2026-05-08T09:20:27.903Z] "config_mode": "enforcing", [2026-05-08T09:20:27.903Z] "mode": "enforcing", [2026-05-08T09:20:27.903Z] "policyvers": 33, [2026-05-08T09:20:27.903Z] "status": "enabled", [2026-05-08T09:20:27.903Z] "type": "targeted" [2026-05-08T09:20:27.903Z] }, [2026-05-08T09:20:27.903Z] "failed_when_result": false [2026-05-08T09:20:27.903Z] } [2026-05-08T09:20:27.903Z] ok: [an-a01n01] => { [2026-05-08T09:20:27.903Z] "ansible_selinux": { [2026-05-08T09:20:27.903Z] "config_mode": "enforcing", [2026-05-08T09:20:27.903Z] "mode": "enforcing", [2026-05-08T09:20:27.903Z] "policyvers": 33, [2026-05-08T09:20:27.903Z] "status": "enabled", [2026-05-08T09:20:27.903Z] "type": "targeted" [2026-05-08T09:20:27.903Z] }, [2026-05-08T09:20:27.903Z] "failed_when_result": false [2026-05-08T09:20:27.903Z] } [2026-05-08T09:20:27.903Z] ok: [an-a01n02] => { [2026-05-08T09:20:27.903Z] "ansible_selinux": { [2026-05-08T09:20:27.903Z] "config_mode": "enforcing", [2026-05-08T09:20:27.903Z] "mode": "enforcing", [2026-05-08T09:20:27.903Z] "policyvers": 33, [2026-05-08T09:20:27.903Z] "status": "enabled", [2026-05-08T09:20:27.903Z] "type": "targeted" [2026-05-08T09:20:27.903Z] }, [2026-05-08T09:20:27.903Z] "failed_when_result": false [2026-05-08T09:20:27.903Z] } [2026-05-08T09:20:27.903Z] [2026-05-08T09:20:27.903Z] PLAY [Create random file on striker01] ***************************************** [2026-05-08T09:20:27.903Z] [2026-05-08T09:20:27.903Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:20:30.330Z] ok: [an-striker01] [2026-05-08T09:20:30.330Z] [2026-05-08T09:20:30.330Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-05-08T09:20:32.051Z] changed: [an-striker01] [2026-05-08T09:20:32.051Z] [2026-05-08T09:20:32.051Z] PLAY [Check archive file propagation] ****************************************** [2026-05-08T09:20:32.051Z] [2026-05-08T09:20:32.051Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:20:33.772Z] ok: [an-a01n02] [2026-05-08T09:20:33.772Z] ok: [an-a01n01] [2026-05-08T09:20:33.772Z] ok: [an-striker01] [2026-05-08T09:20:33.772Z] ok: [an-striker02] [2026-05-08T09:20:33.772Z] [2026-05-08T09:20:33.772Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-05-08T09:20:33.772Z] exit 1 [2026-05-08T09:20:33.772Z] else [2026-05-08T09:20:33.772Z] exit 0 [2026-05-08T09:20:33.772Z] fi [2026-05-08T09:20:33.772Z] ] *** [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-05-08T09:21:57.189Z] ok: [an-striker01] [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-05-08T09:21:57.189Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-05-08T09:21:57.189Z] ok: [an-striker02] [2026-05-08T09:21:57.189Z] ok: [an-a01n02] [2026-05-08T09:21:57.189Z] ok: [an-a01n01] [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] PLAY [test fencing] ************************************************************ [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:21:57.189Z] ok: [an-a01n02] [2026-05-08T09:21:57.189Z] ok: [an-a01n01] [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T09:21:57.189Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T09:21:57.189Z] ] *** [2026-05-08T09:21:57.189Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T09:21:57.189Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T09:21:57.189Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T09:21:57.189Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-05-08T09:21:57.189Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T09:21:57.189Z] ] ********** [2026-05-08T09:21:57.189Z] skipping: [an-a01n02] [2026-05-08T09:21:57.189Z] changed: [an-a01n01] [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T09:21:57.189Z] skipping: [an-a01n02] [2026-05-08T09:21:57.189Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-05-08T09:21:57.189Z] [2026-05-08T09:21:57.189Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T09:25:06.297Z] ok: [an-a01n01 -> localhost] [2026-05-08T09:25:06.297Z] [2026-05-08T09:25:06.297Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T09:25:06.297Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T09:25:06.297Z] [2026-05-08T09:25:06.297Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T09:25:06.297Z] ] *** [2026-05-08T09:25:09.446Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T09:25:09.446Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T09:25:09.446Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T09:25:09.446Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T09:25:09.446Z] [2026-05-08T09:25:09.446Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-05-08T09:25:09.446Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T09:25:09.446Z] [2026-05-08T09:25:09.446Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T09:25:09.446Z] ] ********** [2026-05-08T09:25:10.186Z] skipping: [an-a01n01] [2026-05-08T09:25:10.186Z] changed: [an-a01n02] [2026-05-08T09:25:10.186Z] [2026-05-08T09:25:10.186Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T09:25:10.186Z] skipping: [an-a01n01] [2026-05-08T09:25:10.186Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-05-08T09:25:10.186Z] [2026-05-08T09:25:10.186Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T09:28:34.401Z] ok: [an-a01n02 -> localhost] [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T09:28:34.401Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T09:28:34.401Z] ] *** [2026-05-08T09:28:34.401Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T09:28:34.401Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T09:28:34.401Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T09:28:34.401Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] PLAY [advanced server tests] *************************************************** [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T09:28:34.401Z] ok: [an-a01n01] [2026-05-08T09:28:34.401Z] ok: [an-a01n02] [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T09:28:34.401Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T09:28:34.401Z] ] *** [2026-05-08T09:28:34.401Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T09:28:34.401Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T09:28:34.401Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T09:28:34.401Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-08T09:28:34.401Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-08T09:28:34.401Z] [2026-05-08T09:28:34.401Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-08T09:28:34.401Z] if [ -z "$storage" ]; then [2026-05-08T09:28:34.401Z] echo "Unable to detect shared storage" [2026-05-08T09:28:34.401Z] exit 1 [2026-05-08T09:28:34.401Z] fi [2026-05-08T09:28:34.401Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T09:28:34.401Z] ] *** [2026-05-08T09:28:35.595Z] changed: [an-a01n01] [2026-05-08T09:28:35.595Z] changed: [an-a01n02] [2026-05-08T09:28:35.595Z] [2026-05-08T09:28:35.595Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-08T09:28:35.595Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-08T09:28:35.595Z] [2026-05-08T09:28:35.595Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-08T09:28:35.595Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-08T09:28:35.595Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-08T09:28:35.595Z] echo "Unable to detect all deploy iso" [2026-05-08T09:28:35.595Z] echo "Detected isos" [2026-05-08T09:28:35.595Z] echo "$isos" [2026-05-08T09:28:35.595Z] exit 1 [2026-05-08T09:28:35.595Z] fi [2026-05-08T09:28:35.595Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-08T09:28:35.595Z] if [ -z "$installiso" ]; then [2026-05-08T09:28:35.595Z] echo "Unable to detect install iso" [2026-05-08T09:28:35.595Z] echo "Detected isos" [2026-05-08T09:28:35.595Z] echo "$isos" [2026-05-08T09:28:35.595Z] exit 1 [2026-05-08T09:28:35.595Z] fi [2026-05-08T09:28:35.595Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T09:28:35.595Z] ] *** [2026-05-08T09:28:40.475Z] changed: [an-a01n01] [2026-05-08T09:28:40.475Z] changed: [an-a01n02] [2026-05-08T09:28:40.475Z] [2026-05-08T09:28:40.475Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-05-08T09:28:40.820Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:28:40.820Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:28:40.820Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:28:40.820Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:28:40.820Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:28:40.820Z] [2026-05-08T09:28:40.820Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T09:28:40.820Z] ] *** [2026-05-08T09:28:42.543Z] skipping: [an-a01n02] [2026-05-08T09:28:42.543Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.955885", "end": "2026-05-08 05:28:41.975947", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 05:28:41.020062", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T09:28:42.543Z] ...ignoring [2026-05-08T09:28:42.543Z] [2026-05-08T09:28:42.543Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T09:28:42.543Z] --ci-test \ [2026-05-08T09:28:42.543Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T09:28:42.543Z] --os almalinux9 \ [2026-05-08T09:28:42.543Z] --cpu 2 --ram 4G \ [2026-05-08T09:28:42.543Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T09:28:42.543Z] --install-media {{ installisouuid }} \ [2026-05-08T09:28:42.543Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T09:28:42.543Z] ] *** [2026-05-08T09:28:51.609Z] skipping: [an-a01n02] [2026-05-08T09:28:51.609Z] changed: [an-a01n01] [2026-05-08T09:28:51.609Z] [2026-05-08T09:28:51.609Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T09:28:51.609Z] skipping: [an-a01n01] [2026-05-08T09:28:51.609Z] skipping: [an-a01n02] [2026-05-08T09:28:51.609Z] [2026-05-08T09:28:51.609Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T09:28:51.609Z] skipping: [an-a01n01] [2026-05-08T09:28:51.609Z] skipping: [an-a01n02] [2026-05-08T09:28:51.609Z] [2026-05-08T09:28:51.609Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T09:28:51.609Z] skipping: [an-a01n01] [2026-05-08T09:28:51.609Z] skipping: [an-a01n02] [2026-05-08T09:28:51.609Z] [2026-05-08T09:28:51.609Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T09:28:51.609Z] skipping: [an-a01n01] [2026-05-08T09:28:51.609Z] skipping: [an-a01n02] [2026-05-08T09:28:51.609Z] [2026-05-08T09:28:51.609Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T09:28:51.609Z] ] *** [2026-05-08T09:28:52.349Z] skipping: [an-a01n02] [2026-05-08T09:28:52.349Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.931770", "end": "2026-05-08 05:28:52.213933", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 05:28:51.282163", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T09:28:52.349Z] ...ignoring [2026-05-08T09:28:52.349Z] [2026-05-08T09:28:52.349Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T09:28:52.349Z] --ci-test \ [2026-05-08T09:28:52.349Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T09:28:52.349Z] --os almalinux9 \ [2026-05-08T09:28:52.349Z] --cpu 2 --ram 4G \ [2026-05-08T09:28:52.349Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T09:28:52.349Z] --install-media {{ installisouuid }} \ [2026-05-08T09:28:52.349Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T09:28:52.349Z] ] *** [2026-05-08T09:29:01.510Z] skipping: [an-a01n02] [2026-05-08T09:29:01.510Z] changed: [an-a01n01] [2026-05-08T09:29:01.510Z] [2026-05-08T09:29:01.510Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T09:29:01.510Z] skipping: [an-a01n01] [2026-05-08T09:29:01.510Z] skipping: [an-a01n02] [2026-05-08T09:29:01.510Z] [2026-05-08T09:29:01.510Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T09:29:01.510Z] skipping: [an-a01n01] [2026-05-08T09:29:01.510Z] skipping: [an-a01n02] [2026-05-08T09:29:01.510Z] [2026-05-08T09:29:01.510Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T09:29:01.510Z] skipping: [an-a01n01] [2026-05-08T09:29:01.510Z] skipping: [an-a01n02] [2026-05-08T09:29:01.510Z] [2026-05-08T09:29:01.510Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T09:29:01.510Z] skipping: [an-a01n01] [2026-05-08T09:29:01.510Z] skipping: [an-a01n02] [2026-05-08T09:29:01.510Z] [2026-05-08T09:29:01.510Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T09:29:01.510Z] ] *** [2026-05-08T09:29:02.251Z] skipping: [an-a01n02] [2026-05-08T09:29:02.251Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.829810", "end": "2026-05-08 05:29:02.005212", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 05:29:01.175402", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T09:29:02.251Z] ...ignoring [2026-05-08T09:29:02.251Z] [2026-05-08T09:29:02.251Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T09:29:02.251Z] --ci-test \ [2026-05-08T09:29:02.251Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T09:29:02.251Z] --os almalinux9 \ [2026-05-08T09:29:02.251Z] --cpu 2 --ram 4G \ [2026-05-08T09:29:02.251Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T09:29:02.251Z] --install-media {{ installisouuid }} \ [2026-05-08T09:29:02.251Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T09:29:02.251Z] ] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] changed: [an-a01n01] [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n01] [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n01] [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n01] [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n01] [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T09:29:13.266Z] ] *** [2026-05-08T09:29:13.266Z] skipping: [an-a01n02] [2026-05-08T09:29:13.266Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.885423", "end": "2026-05-08 05:29:13.135599", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 05:29:12.250176", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T09:29:13.266Z] ...ignoring [2026-05-08T09:29:13.266Z] [2026-05-08T09:29:13.266Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T09:29:13.266Z] --ci-test \ [2026-05-08T09:29:13.266Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T09:29:13.266Z] --os almalinux9 \ [2026-05-08T09:29:13.266Z] --cpu 2 --ram 4G \ [2026-05-08T09:29:13.266Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T09:29:13.266Z] --install-media {{ installisouuid }} \ [2026-05-08T09:29:13.266Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T09:29:13.266Z] ] *** [2026-05-08T09:29:22.333Z] skipping: [an-a01n02] [2026-05-08T09:29:22.333Z] changed: [an-a01n01] [2026-05-08T09:29:22.333Z] [2026-05-08T09:29:22.333Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T09:29:22.333Z] skipping: [an-a01n01] [2026-05-08T09:29:22.333Z] skipping: [an-a01n02] [2026-05-08T09:29:22.333Z] [2026-05-08T09:29:22.333Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T09:29:22.333Z] skipping: [an-a01n01] [2026-05-08T09:29:22.333Z] skipping: [an-a01n02] [2026-05-08T09:29:22.333Z] [2026-05-08T09:29:22.333Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T09:29:22.333Z] skipping: [an-a01n01] [2026-05-08T09:29:22.333Z] skipping: [an-a01n02] [2026-05-08T09:29:22.333Z] [2026-05-08T09:29:22.333Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T09:29:22.333Z] skipping: [an-a01n01] [2026-05-08T09:29:22.333Z] skipping: [an-a01n02] [2026-05-08T09:29:22.333Z] [2026-05-08T09:29:22.333Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T09:29:22.333Z] ] *** [2026-05-08T09:29:24.055Z] skipping: [an-a01n02] [2026-05-08T09:29:24.055Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.950013", "end": "2026-05-08 05:29:23.830542", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 05:29:22.880529", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T09:29:24.055Z] ...ignoring [2026-05-08T09:29:24.055Z] [2026-05-08T09:29:24.055Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T09:29:24.055Z] --ci-test \ [2026-05-08T09:29:24.055Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T09:29:24.055Z] --os almalinux9 \ [2026-05-08T09:29:24.055Z] --cpu 2 --ram 4G \ [2026-05-08T09:29:24.055Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T09:29:24.055Z] --install-media {{ installisouuid }} \ [2026-05-08T09:29:24.055Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T09:29:24.055Z] ] *** [2026-05-08T09:29:35.164Z] skipping: [an-a01n02] [2026-05-08T09:29:35.164Z] changed: [an-a01n01] [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T09:29:35.164Z] skipping: [an-a01n01] [2026-05-08T09:29:35.164Z] skipping: [an-a01n02] [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T09:29:35.164Z] skipping: [an-a01n01] [2026-05-08T09:29:35.164Z] skipping: [an-a01n02] [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T09:29:35.164Z] skipping: [an-a01n01] [2026-05-08T09:29:35.164Z] skipping: [an-a01n02] [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T09:29:35.164Z] skipping: [an-a01n01] [2026-05-08T09:29:35.164Z] skipping: [an-a01n02] [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-05-08T09:29:35.164Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:29:35.164Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:29:35.164Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:29:35.164Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:29:35.164Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:29:35.164Z] [2026-05-08T09:29:35.164Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T09:29:35.164Z] ] *** [2026-05-08T09:30:03.180Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-05-08T09:30:03.180Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-05-08T09:30:03.180Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-05-08T09:30:03.180Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-05-08T09:30:03.180Z] ok: [an-a01n01] [2026-05-08T09:30:03.180Z] ok: [an-a01n02] [2026-05-08T09:30:03.180Z] [2026-05-08T09:30:03.180Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T09:30:03.180Z] ] *** [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-05-08T09:32:11.986Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-05-08T09:32:11.986Z] ok: [an-a01n02] [2026-05-08T09:32:11.986Z] ok: [an-a01n01] [2026-05-08T09:32:11.986Z] [2026-05-08T09:32:11.986Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T09:32:11.986Z] ] *** [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-05-08T09:34:35.920Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-05-08T09:34:35.920Z] ok: [an-a01n02] [2026-05-08T09:34:35.920Z] ok: [an-a01n01] [2026-05-08T09:34:35.920Z] [2026-05-08T09:34:35.920Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T09:34:35.920Z] ] *** [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-05-08T09:37:14.919Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-05-08T09:37:14.919Z] ok: [an-a01n02] [2026-05-08T09:37:14.919Z] ok: [an-a01n01] [2026-05-08T09:37:14.919Z] [2026-05-08T09:37:14.919Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T09:37:14.919Z] ] *** [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-05-08T09:40:39.139Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-05-08T09:40:39.139Z] ok: [an-a01n02] [2026-05-08T09:40:39.139Z] ok: [an-a01n01] [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-05-08T09:40:39.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:40:39.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:40:39.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:40:39.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:40:39.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T09:40:39.139Z] ] *** [2026-05-08T09:40:39.139Z] ok: [an-a01n02] [2026-05-08T09:40:39.139Z] ok: [an-a01n01] [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T09:40:39.139Z] ] *** [2026-05-08T09:40:39.139Z] ok: [an-a01n02] [2026-05-08T09:40:39.139Z] ok: [an-a01n01] [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T09:40:39.139Z] ] *** [2026-05-08T09:40:39.139Z] ok: [an-a01n02] [2026-05-08T09:40:39.139Z] ok: [an-a01n01] [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T09:40:39.139Z] ] *** [2026-05-08T09:40:39.139Z] ok: [an-a01n02] [2026-05-08T09:40:39.139Z] ok: [an-a01n01] [2026-05-08T09:40:39.139Z] [2026-05-08T09:40:39.139Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T09:40:39.139Z] ] *** [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-05-08T09:41:07.250Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-05-08T09:41:07.250Z] ok: [an-a01n02] [2026-05-08T09:41:07.250Z] ok: [an-a01n01] [2026-05-08T09:41:07.250Z] [2026-05-08T09:41:07.250Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-05-08T09:41:07.250Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:41:07.250Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:41:07.250Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:41:07.250Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:41:07.250Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:41:07.250Z] [2026-05-08T09:41:07.250Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T09:41:07.250Z] ] *** [2026-05-08T09:41:07.250Z] ok: [an-a01n02] [2026-05-08T09:41:07.250Z] ok: [an-a01n01] [2026-05-08T09:41:07.250Z] [2026-05-08T09:41:07.250Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T09:41:07.250Z] ] *** [2026-05-08T09:41:07.250Z] ok: [an-a01n02] [2026-05-08T09:41:07.250Z] ok: [an-a01n01] [2026-05-08T09:41:07.250Z] [2026-05-08T09:41:07.250Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T09:41:07.250Z] ] *** [2026-05-08T09:42:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-05-08T09:42:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-05-08T09:42:05.394Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-05-08T09:42:05.394Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-05-08T09:42:05.395Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-05-08T09:42:05.395Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-05-08T09:42:05.395Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-05-08T09:42:05.395Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-05-08T09:42:05.395Z] ok: [an-a01n02] [2026-05-08T09:42:05.395Z] ok: [an-a01n01] [2026-05-08T09:42:05.395Z] [2026-05-08T09:42:05.395Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T09:42:05.395Z] ] *** [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-05-08T09:45:14.517Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-05-08T09:45:14.517Z] ok: [an-a01n02] [2026-05-08T09:45:14.517Z] ok: [an-a01n01] [2026-05-08T09:45:14.517Z] [2026-05-08T09:45:14.517Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T09:45:14.517Z] ] *** [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-05-08T09:47:53.438Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-05-08T09:47:53.438Z] ok: [an-a01n02] [2026-05-08T09:47:53.438Z] ok: [an-a01n01] [2026-05-08T09:47:53.438Z] [2026-05-08T09:47:53.438Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-05-08T09:47:53.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:47:53.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:47:53.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:47:53.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:47:53.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:47:53.438Z] [2026-05-08T09:47:53.438Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:47:53.438Z] ] *** [2026-05-08T09:47:54.179Z] ok: [an-a01n02] [2026-05-08T09:47:54.179Z] ok: [an-a01n01] [2026-05-08T09:47:54.179Z] [2026-05-08T09:47:54.179Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:47:54.179Z] ] *** [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-05-08T09:49:32.776Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-05-08T09:49:32.776Z] ok: [an-a01n02] [2026-05-08T09:49:32.776Z] ok: [an-a01n01] [2026-05-08T09:49:32.776Z] [2026-05-08T09:49:32.776Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:49:32.776Z] ] *** [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-05-08T09:51:56.593Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-05-08T09:51:56.593Z] ok: [an-a01n02] [2026-05-08T09:51:56.593Z] ok: [an-a01n01] [2026-05-08T09:51:56.593Z] [2026-05-08T09:51:56.593Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:51:56.593Z] ] *** [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-05-08T09:54:05.403Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-05-08T09:54:05.403Z] ok: [an-a01n01] [2026-05-08T09:54:05.403Z] ok: [an-a01n02] [2026-05-08T09:54:05.403Z] [2026-05-08T09:54:05.403Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:54:05.403Z] ] *** [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-05-08T09:56:14.130Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-05-08T09:56:14.130Z] ok: [an-a01n02] [2026-05-08T09:56:14.130Z] ok: [an-a01n01] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-05-08T09:56:14.130Z] ok: [an-a01n01] [2026-05-08T09:56:14.130Z] ok: [an-a01n02] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T09:56:14.130Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T09:56:14.130Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T09:56:14.130Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T09:56:14.130Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T09:56:14.130Z] echo "pcs:" [2026-05-08T09:56:14.130Z] echo "$pcsstatus" [2026-05-08T09:56:14.130Z] exit 1 [2026-05-08T09:56:14.130Z] fi [2026-05-08T09:56:14.130Z] exit 0 [2026-05-08T09:56:14.130Z] ] *** [2026-05-08T09:56:14.130Z] ok: [an-a01n01] [2026-05-08T09:56:14.130Z] ok: [an-a01n02] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T09:56:14.130Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T09:56:14.130Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T09:56:14.130Z] ret=0 [2026-05-08T09:56:14.130Z] if [ "$servers" != "1" ]; then [2026-05-08T09:56:14.130Z] echo "Not all servers are running on the same node!" [2026-05-08T09:56:14.130Z] echo "$pcsstatus" [2026-05-08T09:56:14.130Z] ret=1 [2026-05-08T09:56:14.130Z] fi [2026-05-08T09:56:14.130Z] echo "$hypervisor" [2026-05-08T09:56:14.130Z] exit $ret [2026-05-08T09:56:14.130Z] ] *** [2026-05-08T09:56:14.130Z] ok: [an-a01n01] [2026-05-08T09:56:14.130Z] ok: [an-a01n02] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-08T09:56:14.130Z] ok: [an-a01n01] [2026-05-08T09:56:14.130Z] ok: [an-a01n02] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T09:56:14.130Z] ] ********** [2026-05-08T09:56:14.130Z] skipping: [an-a01n02] [2026-05-08T09:56:14.130Z] changed: [an-a01n01] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T09:56:14.130Z] skipping: [an-a01n01] [2026-05-08T09:56:14.130Z] skipping: [an-a01n02] [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-08T09:56:14.130Z] skipping: [an-a01n01] [2026-05-08T09:56:14.130Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-08T09:56:14.130Z] [2026-05-08T09:56:14.130Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-08T09:56:14.130Z] ] *** [2026-05-08T09:57:23.719Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-08T09:57:23.719Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-08T09:57:23.719Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-08T09:57:23.719Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-08T09:57:23.719Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-08T09:57:23.719Z] ok: [an-a01n02] [2026-05-08T09:57:23.719Z] [2026-05-08T09:57:23.719Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T09:57:23.719Z] skipping: [an-a01n01] [2026-05-08T09:57:23.719Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-08T09:57:23.719Z] [2026-05-08T09:57:23.719Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T09:57:23.719Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T09:57:23.719Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T09:57:23.719Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T09:57:23.719Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T09:57:23.719Z] echo "pcs:" [2026-05-08T09:57:23.719Z] echo "$pcsstatus" [2026-05-08T09:57:23.719Z] exit 1 [2026-05-08T09:57:23.719Z] fi [2026-05-08T09:57:23.719Z] exit 0 [2026-05-08T09:57:23.719Z] ] *** [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-05-08T09:58:47.441Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-05-08T09:58:47.441Z] ok: [an-a01n02] [2026-05-08T09:58:47.441Z] [2026-05-08T09:58:47.441Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T09:58:47.441Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T09:58:47.441Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T09:58:47.441Z] ret=0 [2026-05-08T09:58:47.441Z] if [ "$servers" != "1" ]; then [2026-05-08T09:58:47.441Z] echo "Not all servers are running on the same node!" [2026-05-08T09:58:47.441Z] echo "$pcsstatus" [2026-05-08T09:58:47.441Z] ret=1 [2026-05-08T09:58:47.441Z] fi [2026-05-08T09:58:47.441Z] echo "$hypervisor" [2026-05-08T09:58:47.441Z] exit $ret [2026-05-08T09:58:47.441Z] ] *** [2026-05-08T09:58:47.441Z] ok: [an-a01n02] [2026-05-08T09:58:47.441Z] [2026-05-08T09:58:47.441Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] => (item=1) [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] => (item=2) [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] => (item=3) [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] => (item=4) [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] => (item=5) [2026-05-08T09:58:47.441Z] skipping: [an-a01n01] [2026-05-08T09:58:47.441Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-08T09:58:47.441Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-08T09:58:47.441Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-08T09:58:47.441Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-08T09:58:47.441Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-08T09:58:47.441Z] [2026-05-08T09:58:47.441Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:58:47.441Z] ] *** [2026-05-08T09:58:47.441Z] ok: [an-a01n02] [2026-05-08T09:58:47.441Z] [2026-05-08T09:58:47.441Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:58:47.441Z] ] *** [2026-05-08T09:58:47.441Z] ok: [an-a01n02] [2026-05-08T09:58:47.441Z] [2026-05-08T09:58:47.441Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:58:47.441Z] ] *** [2026-05-08T09:58:47.786Z] ok: [an-a01n02] [2026-05-08T09:58:47.786Z] [2026-05-08T09:58:47.786Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:58:47.786Z] ] *** [2026-05-08T09:58:48.526Z] ok: [an-a01n02] [2026-05-08T09:58:48.526Z] [2026-05-08T09:58:48.526Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T09:58:48.526Z] ] *** [2026-05-08T09:58:49.267Z] ok: [an-a01n02] [2026-05-08T09:58:49.267Z] [2026-05-08T09:58:49.267Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-08T09:58:49.267Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-08T09:58:49.267Z] [2026-05-08T09:58:49.267Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T10:02:13.457Z] ok: [an-a01n02 -> localhost] [2026-05-08T10:02:13.457Z] ok: [an-a01n01 -> localhost] [2026-05-08T10:02:13.457Z] [2026-05-08T10:02:13.457Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:02:13.457Z] [2026-05-08T10:02:13.457Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:02:13.457Z] ] *** [2026-05-08T10:02:13.457Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:02:13.457Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:02:13.457Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:02:13.457Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:02:13.457Z] [2026-05-08T10:02:13.457Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:02:13.457Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:02:13.457Z] [2026-05-08T10:02:13.457Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:02:13.457Z] ] *** [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-08T10:03:11.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-05-08T10:03:11.501Z] ok: [an-a01n01] [2026-05-08T10:03:11.501Z] ok: [an-a01n02] [2026-05-08T10:03:11.501Z] [2026-05-08T10:03:11.501Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:03:11.501Z] ] *** [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (358 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (357 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (356 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (355 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (354 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (353 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left). [2026-05-08T10:04:50.060Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (352 retries left). [2026-05-08T10:04:50.060Z] ok: [an-a01n02] [2026-05-08T10:04:50.060Z] ok: [an-a01n01] [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:04:50.060Z] ] *** [2026-05-08T10:04:50.060Z] ok: [an-a01n02] [2026-05-08T10:04:50.060Z] ok: [an-a01n01] [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:04:50.060Z] ] *** [2026-05-08T10:04:50.060Z] ok: [an-a01n01] [2026-05-08T10:04:50.060Z] ok: [an-a01n02] [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:04:50.060Z] ] *** [2026-05-08T10:04:50.060Z] ok: [an-a01n02] [2026-05-08T10:04:50.060Z] ok: [an-a01n01] [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-05-08T10:04:50.060Z] ok: [an-a01n01] [2026-05-08T10:04:50.060Z] ok: [an-a01n02] [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:04:50.060Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:04:50.060Z] [2026-05-08T10:04:50.060Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:04:50.060Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:04:50.060Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:04:50.060Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:04:50.060Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:04:50.060Z] echo "pcs:" [2026-05-08T10:04:50.060Z] echo "$pcsstatus" [2026-05-08T10:04:50.060Z] exit 1 [2026-05-08T10:04:50.060Z] fi [2026-05-08T10:04:50.060Z] exit 0 [2026-05-08T10:04:50.060Z] ] *** [2026-05-08T10:04:52.394Z] ok: [an-a01n02] [2026-05-08T10:04:52.394Z] ok: [an-a01n01] [2026-05-08T10:04:52.394Z] [2026-05-08T10:04:52.394Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:04:52.394Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:04:52.394Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:04:52.394Z] ret=0 [2026-05-08T10:04:52.394Z] if [ "$servers" != "1" ]; then [2026-05-08T10:04:52.394Z] echo "Not all servers are running on the same node!" [2026-05-08T10:04:52.394Z] echo "$pcsstatus" [2026-05-08T10:04:52.394Z] ret=1 [2026-05-08T10:04:52.394Z] fi [2026-05-08T10:04:52.394Z] echo "$hypervisor" [2026-05-08T10:04:52.394Z] exit $ret [2026-05-08T10:04:52.394Z] ] *** [2026-05-08T10:04:54.116Z] ok: [an-a01n01] [2026-05-08T10:04:54.116Z] ok: [an-a01n02] [2026-05-08T10:04:54.116Z] [2026-05-08T10:04:54.116Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-08T10:04:54.116Z] ok: [an-a01n01] [2026-05-08T10:04:54.116Z] ok: [an-a01n02] [2026-05-08T10:04:54.116Z] [2026-05-08T10:04:54.116Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-08T10:04:54.116Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:04:54.116Z] [2026-05-08T10:04:54.116Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T10:04:54.116Z] ] ********** [2026-05-08T10:04:54.855Z] skipping: [an-a01n01] [2026-05-08T10:04:54.855Z] changed: [an-a01n02] [2026-05-08T10:04:54.855Z] [2026-05-08T10:04:54.855Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T10:04:54.855Z] skipping: [an-a01n01] [2026-05-08T10:04:54.855Z] skipping: [an-a01n02] [2026-05-08T10:04:54.855Z] [2026-05-08T10:04:54.855Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-08T10:04:54.855Z] skipping: [an-a01n02] [2026-05-08T10:04:54.855Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-05-08T10:04:54.855Z] [2026-05-08T10:04:54.855Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-08T10:04:54.855Z] ] *** [2026-05-08T10:06:04.586Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-05-08T10:06:04.586Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-05-08T10:06:04.586Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-05-08T10:06:04.586Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-05-08T10:06:04.586Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-05-08T10:06:04.586Z] ok: [an-a01n01] [2026-05-08T10:06:04.586Z] [2026-05-08T10:06:04.586Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:06:04.586Z] skipping: [an-a01n02] [2026-05-08T10:06:04.586Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-05-08T10:06:04.586Z] [2026-05-08T10:06:04.586Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:06:04.586Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:06:04.586Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:06:04.586Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:06:04.586Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:06:04.586Z] echo "pcs:" [2026-05-08T10:06:04.586Z] echo "$pcsstatus" [2026-05-08T10:06:04.586Z] exit 1 [2026-05-08T10:06:04.586Z] fi [2026-05-08T10:06:04.586Z] exit 0 [2026-05-08T10:06:04.586Z] ] *** [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-08T10:07:14.170Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-05-08T10:07:14.170Z] ok: [an-a01n01] [2026-05-08T10:07:14.170Z] [2026-05-08T10:07:14.170Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:07:14.170Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:07:14.170Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:07:14.170Z] ret=0 [2026-05-08T10:07:14.170Z] if [ "$servers" != "1" ]; then [2026-05-08T10:07:14.170Z] echo "Not all servers are running on the same node!" [2026-05-08T10:07:14.170Z] echo "$pcsstatus" [2026-05-08T10:07:14.170Z] ret=1 [2026-05-08T10:07:14.170Z] fi [2026-05-08T10:07:14.170Z] echo "$hypervisor" [2026-05-08T10:07:14.170Z] exit $ret [2026-05-08T10:07:14.170Z] ] *** [2026-05-08T10:07:14.170Z] ok: [an-a01n01] [2026-05-08T10:07:14.170Z] [2026-05-08T10:07:14.170Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] => (item=1) [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] => (item=2) [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] => (item=3) [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] => (item=4) [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] => (item=5) [2026-05-08T10:07:14.170Z] skipping: [an-a01n02] [2026-05-08T10:07:14.170Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-05-08T10:07:14.170Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-05-08T10:07:14.170Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-05-08T10:07:14.170Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-05-08T10:07:14.170Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-05-08T10:07:14.170Z] [2026-05-08T10:07:14.170Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:07:14.170Z] ] *** [2026-05-08T10:07:15.366Z] ok: [an-a01n01] [2026-05-08T10:07:15.366Z] [2026-05-08T10:07:15.366Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:07:15.366Z] ] *** [2026-05-08T10:07:16.244Z] ok: [an-a01n01] [2026-05-08T10:07:16.244Z] [2026-05-08T10:07:16.244Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:07:16.244Z] ] *** [2026-05-08T10:07:16.984Z] ok: [an-a01n01] [2026-05-08T10:07:16.984Z] [2026-05-08T10:07:16.984Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:07:16.984Z] ] *** [2026-05-08T10:07:18.194Z] ok: [an-a01n01] [2026-05-08T10:07:18.194Z] [2026-05-08T10:07:18.194Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:07:18.194Z] ] *** [2026-05-08T10:07:19.388Z] ok: [an-a01n01] [2026-05-08T10:07:19.388Z] [2026-05-08T10:07:19.388Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-08T10:07:19.388Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:07:19.388Z] [2026-05-08T10:07:19.388Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T10:10:43.734Z] ok: [an-a01n01 -> localhost] [2026-05-08T10:10:43.734Z] ok: [an-a01n02 -> localhost] [2026-05-08T10:10:43.734Z] [2026-05-08T10:10:43.734Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:10:43.734Z] [2026-05-08T10:10:43.734Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:10:43.734Z] ] *** [2026-05-08T10:10:43.734Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:10:43.734Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:10:43.734Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:10:43.734Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:10:43.734Z] [2026-05-08T10:10:43.734Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:10:43.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:10:43.734Z] [2026-05-08T10:10:43.734Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:10:43.734Z] ] *** [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-08T10:12:22.256Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-05-08T10:12:22.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:12:22.257Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:12:22.257Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:12:22.257Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:12:22.257Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:12:22.257Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:12:22.257Z] echo "pcs:" [2026-05-08T10:12:22.257Z] echo "$pcsstatus" [2026-05-08T10:12:22.257Z] exit 1 [2026-05-08T10:12:22.257Z] fi [2026-05-08T10:12:22.257Z] exit 0 [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:12:22.257Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:12:22.257Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:12:22.257Z] ret=0 [2026-05-08T10:12:22.257Z] if [ "$servers" != "1" ]; then [2026-05-08T10:12:22.257Z] echo "Not all servers are running on the same node!" [2026-05-08T10:12:22.257Z] echo "$pcsstatus" [2026-05-08T10:12:22.257Z] ret=1 [2026-05-08T10:12:22.257Z] fi [2026-05-08T10:12:22.257Z] echo "$hypervisor" [2026-05-08T10:12:22.257Z] exit $ret [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-08T10:12:22.257Z] ok: [an-a01n01] [2026-05-08T10:12:22.257Z] ok: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-08T10:12:22.257Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T10:12:22.257Z] ] ********** [2026-05-08T10:12:22.257Z] skipping: [an-a01n02] [2026-05-08T10:12:22.257Z] changed: [an-a01n01] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T10:12:22.257Z] skipping: [an-a01n01] [2026-05-08T10:12:22.257Z] skipping: [an-a01n02] [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-08T10:12:22.257Z] skipping: [an-a01n01] [2026-05-08T10:12:22.257Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-08T10:12:22.257Z] [2026-05-08T10:12:22.257Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-08T10:12:22.257Z] ] *** [2026-05-08T10:13:32.001Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-08T10:13:32.001Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-08T10:13:32.001Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-08T10:13:32.001Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-08T10:13:32.001Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-08T10:13:32.001Z] ok: [an-a01n02] [2026-05-08T10:13:32.001Z] [2026-05-08T10:13:32.001Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:13:32.001Z] skipping: [an-a01n01] [2026-05-08T10:13:32.001Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-08T10:13:32.001Z] [2026-05-08T10:13:32.001Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:13:32.001Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:13:32.001Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:13:32.001Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:13:32.001Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:13:32.001Z] echo "pcs:" [2026-05-08T10:13:32.001Z] echo "$pcsstatus" [2026-05-08T10:13:32.001Z] exit 1 [2026-05-08T10:13:32.001Z] fi [2026-05-08T10:13:32.001Z] exit 0 [2026-05-08T10:13:32.001Z] ] *** [2026-05-08T10:14:12.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-08T10:14:12.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-08T10:14:12.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-08T10:14:12.362Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-08T10:14:12.362Z] ok: [an-a01n02] [2026-05-08T10:14:12.362Z] [2026-05-08T10:14:12.362Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:14:12.362Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:14:12.362Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:14:12.362Z] ret=0 [2026-05-08T10:14:12.362Z] if [ "$servers" != "1" ]; then [2026-05-08T10:14:12.362Z] echo "Not all servers are running on the same node!" [2026-05-08T10:14:12.362Z] echo "$pcsstatus" [2026-05-08T10:14:12.362Z] ret=1 [2026-05-08T10:14:12.362Z] fi [2026-05-08T10:14:12.362Z] echo "$hypervisor" [2026-05-08T10:14:12.362Z] exit $ret [2026-05-08T10:14:12.362Z] ] *** [2026-05-08T10:14:14.701Z] ok: [an-a01n02] [2026-05-08T10:14:14.701Z] [2026-05-08T10:14:14.701Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] => (item=1) [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] => (item=2) [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] => (item=3) [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] => (item=4) [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] => (item=5) [2026-05-08T10:14:14.701Z] skipping: [an-a01n01] [2026-05-08T10:14:14.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-08T10:14:14.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-08T10:14:14.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-08T10:14:14.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-08T10:14:14.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-08T10:14:14.701Z] [2026-05-08T10:14:14.701Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:14:14.701Z] ] *** [2026-05-08T10:14:30.796Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-08T10:14:30.796Z] ok: [an-a01n02] [2026-05-08T10:14:30.796Z] [2026-05-08T10:14:30.796Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:14:30.796Z] ] *** [2026-05-08T10:14:44.132Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-05-08T10:14:44.132Z] ok: [an-a01n02] [2026-05-08T10:14:44.132Z] [2026-05-08T10:14:44.132Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:14:44.132Z] ] *** [2026-05-08T10:14:45.022Z] ok: [an-a01n02] [2026-05-08T10:14:45.022Z] [2026-05-08T10:14:45.022Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:14:45.022Z] ] *** [2026-05-08T10:14:45.763Z] ok: [an-a01n02] [2026-05-08T10:14:45.763Z] [2026-05-08T10:14:45.763Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:14:45.763Z] ] *** [2026-05-08T10:14:46.504Z] ok: [an-a01n02] [2026-05-08T10:14:46.504Z] [2026-05-08T10:14:46.504Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-08T10:14:46.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:14:46.504Z] [2026-05-08T10:14:46.504Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01 -> localhost] [2026-05-08T10:18:10.760Z] ok: [an-a01n02 -> localhost] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:18:10.760Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:18:10.760Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:18:10.760Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:18:10.760Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:18:10.760Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:18:10.760Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:18:10.760Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:18:10.760Z] echo "pcs:" [2026-05-08T10:18:10.760Z] echo "$pcsstatus" [2026-05-08T10:18:10.760Z] exit 1 [2026-05-08T10:18:10.760Z] fi [2026-05-08T10:18:10.760Z] exit 0 [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:18:10.760Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:18:10.760Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:18:10.760Z] ret=0 [2026-05-08T10:18:10.760Z] if [ "$servers" != "1" ]; then [2026-05-08T10:18:10.760Z] echo "Not all servers are running on the same node!" [2026-05-08T10:18:10.760Z] echo "$pcsstatus" [2026-05-08T10:18:10.760Z] ret=1 [2026-05-08T10:18:10.760Z] fi [2026-05-08T10:18:10.760Z] echo "$hypervisor" [2026-05-08T10:18:10.760Z] exit $ret [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-08T10:18:10.760Z] ok: [an-a01n01] [2026-05-08T10:18:10.760Z] ok: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T10:18:10.760Z] ] ********** [2026-05-08T10:18:10.760Z] skipping: [an-a01n01] [2026-05-08T10:18:10.760Z] changed: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T10:18:10.760Z] skipping: [an-a01n01] [2026-05-08T10:18:10.760Z] skipping: [an-a01n02] [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-08T10:18:10.760Z] skipping: [an-a01n02] [2026-05-08T10:18:10.760Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-05-08T10:18:10.760Z] [2026-05-08T10:18:10.760Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-08T10:18:10.760Z] ] *** [2026-05-08T10:19:20.605Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-05-08T10:19:20.605Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-05-08T10:19:20.605Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-05-08T10:19:20.605Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-05-08T10:19:20.605Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-05-08T10:19:20.605Z] ok: [an-a01n01] [2026-05-08T10:19:20.605Z] [2026-05-08T10:19:20.605Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:19:20.605Z] skipping: [an-a01n02] [2026-05-08T10:19:20.605Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-05-08T10:19:20.605Z] [2026-05-08T10:19:20.605Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:19:20.605Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:19:20.605Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:19:20.605Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:19:20.605Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:19:20.605Z] echo "pcs:" [2026-05-08T10:19:20.605Z] echo "$pcsstatus" [2026-05-08T10:19:20.605Z] exit 1 [2026-05-08T10:19:20.605Z] fi [2026-05-08T10:19:20.605Z] exit 0 [2026-05-08T10:19:20.605Z] ] *** [2026-05-08T10:20:18.883Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-08T10:20:18.883Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-08T10:20:18.883Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-08T10:20:18.883Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-08T10:20:18.883Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-08T10:20:18.883Z] ok: [an-a01n01] [2026-05-08T10:20:18.883Z] [2026-05-08T10:20:18.883Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:20:18.883Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:20:18.883Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:20:18.883Z] ret=0 [2026-05-08T10:20:18.883Z] if [ "$servers" != "1" ]; then [2026-05-08T10:20:18.883Z] echo "Not all servers are running on the same node!" [2026-05-08T10:20:18.883Z] echo "$pcsstatus" [2026-05-08T10:20:18.883Z] ret=1 [2026-05-08T10:20:18.883Z] fi [2026-05-08T10:20:18.883Z] echo "$hypervisor" [2026-05-08T10:20:18.883Z] exit $ret [2026-05-08T10:20:18.883Z] ] *** [2026-05-08T10:20:18.883Z] ok: [an-a01n01] [2026-05-08T10:20:18.883Z] [2026-05-08T10:20:18.883Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] => (item=1) [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] => (item=2) [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] => (item=3) [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] => (item=4) [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] => (item=5) [2026-05-08T10:20:18.883Z] skipping: [an-a01n02] [2026-05-08T10:20:18.883Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-05-08T10:20:18.883Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-05-08T10:20:18.883Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-05-08T10:20:18.883Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-05-08T10:20:18.883Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-05-08T10:20:18.883Z] [2026-05-08T10:20:18.883Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:20:18.883Z] ] *** [2026-05-08T10:20:30.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-08T10:20:30.948Z] ok: [an-a01n01] [2026-05-08T10:20:30.948Z] [2026-05-08T10:20:30.948Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:20:30.948Z] ] *** [2026-05-08T10:20:31.690Z] ok: [an-a01n01] [2026-05-08T10:20:31.690Z] [2026-05-08T10:20:31.690Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:20:31.690Z] ] *** [2026-05-08T10:20:32.430Z] ok: [an-a01n01] [2026-05-08T10:20:32.430Z] [2026-05-08T10:20:32.430Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:20:32.430Z] ] *** [2026-05-08T10:20:33.170Z] ok: [an-a01n01] [2026-05-08T10:20:33.170Z] [2026-05-08T10:20:33.170Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:20:33.170Z] ] *** [2026-05-08T10:20:33.911Z] ok: [an-a01n01] [2026-05-08T10:20:33.911Z] [2026-05-08T10:20:33.911Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-08T10:20:33.911Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:20:33.911Z] [2026-05-08T10:20:33.911Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n01 -> localhost] [2026-05-08T10:23:58.243Z] ok: [an-a01n02 -> localhost] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:23:58.243Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:23:58.243Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:23:58.243Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:23:58.243Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:23:58.243Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:23:58.243Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:23:58.243Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:23:58.243Z] echo "pcs:" [2026-05-08T10:23:58.243Z] echo "$pcsstatus" [2026-05-08T10:23:58.243Z] exit 1 [2026-05-08T10:23:58.243Z] fi [2026-05-08T10:23:58.243Z] exit 0 [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:23:58.243Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:23:58.243Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:23:58.243Z] ret=0 [2026-05-08T10:23:58.243Z] if [ "$servers" != "1" ]; then [2026-05-08T10:23:58.243Z] echo "Not all servers are running on the same node!" [2026-05-08T10:23:58.243Z] echo "$pcsstatus" [2026-05-08T10:23:58.243Z] ret=1 [2026-05-08T10:23:58.243Z] fi [2026-05-08T10:23:58.243Z] echo "$hypervisor" [2026-05-08T10:23:58.243Z] exit $ret [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-05-08T10:23:58.243Z] ok: [an-a01n01] [2026-05-08T10:23:58.243Z] ok: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-05-08T10:23:58.243Z] ] ********** [2026-05-08T10:23:58.243Z] skipping: [an-a01n02] [2026-05-08T10:23:58.243Z] changed: [an-a01n01] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-05-08T10:23:58.243Z] skipping: [an-a01n01] [2026-05-08T10:23:58.243Z] skipping: [an-a01n02] [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-05-08T10:23:58.243Z] skipping: [an-a01n01] [2026-05-08T10:23:58.243Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-05-08T10:23:58.243Z] [2026-05-08T10:23:58.243Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-05-08T10:23:58.243Z] ] *** [2026-05-08T10:25:07.853Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-05-08T10:25:07.853Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-05-08T10:25:07.853Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-05-08T10:25:07.853Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-05-08T10:25:07.853Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-05-08T10:25:07.853Z] ok: [an-a01n02] [2026-05-08T10:25:07.853Z] [2026-05-08T10:25:07.853Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-05-08T10:25:07.853Z] skipping: [an-a01n01] [2026-05-08T10:25:07.853Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-05-08T10:25:07.853Z] [2026-05-08T10:25:07.853Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:25:07.853Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:25:07.853Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:25:07.853Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:25:07.853Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:25:07.853Z] echo "pcs:" [2026-05-08T10:25:07.853Z] echo "$pcsstatus" [2026-05-08T10:25:07.853Z] exit 1 [2026-05-08T10:25:07.853Z] fi [2026-05-08T10:25:07.853Z] exit 0 [2026-05-08T10:25:07.853Z] ] *** [2026-05-08T10:26:05.976Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-05-08T10:26:05.976Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-05-08T10:26:05.976Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-05-08T10:26:05.976Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-05-08T10:26:05.976Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-05-08T10:26:05.976Z] ok: [an-a01n02] [2026-05-08T10:26:05.976Z] [2026-05-08T10:26:05.976Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:26:05.976Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:26:05.976Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:26:05.976Z] ret=0 [2026-05-08T10:26:05.976Z] if [ "$servers" != "1" ]; then [2026-05-08T10:26:05.976Z] echo "Not all servers are running on the same node!" [2026-05-08T10:26:05.976Z] echo "$pcsstatus" [2026-05-08T10:26:05.976Z] ret=1 [2026-05-08T10:26:05.976Z] fi [2026-05-08T10:26:05.976Z] echo "$hypervisor" [2026-05-08T10:26:05.976Z] exit $ret [2026-05-08T10:26:05.976Z] ] *** [2026-05-08T10:26:05.976Z] ok: [an-a01n02] [2026-05-08T10:26:05.976Z] [2026-05-08T10:26:05.976Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] => (item=1) [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] => (item=2) [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] => (item=3) [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] => (item=4) [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] => (item=5) [2026-05-08T10:26:05.976Z] skipping: [an-a01n01] [2026-05-08T10:26:05.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-05-08T10:26:05.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-05-08T10:26:05.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-05-08T10:26:05.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-05-08T10:26:05.976Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-05-08T10:26:05.976Z] [2026-05-08T10:26:05.976Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:26:05.976Z] ] *** [2026-05-08T10:26:19.740Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-05-08T10:26:19.740Z] ok: [an-a01n02] [2026-05-08T10:26:19.740Z] [2026-05-08T10:26:19.740Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:26:19.740Z] ] *** [2026-05-08T10:26:19.740Z] ok: [an-a01n02] [2026-05-08T10:26:19.740Z] [2026-05-08T10:26:19.740Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:26:19.740Z] ] *** [2026-05-08T10:26:20.085Z] ok: [an-a01n02] [2026-05-08T10:26:20.085Z] [2026-05-08T10:26:20.085Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:26:20.085Z] ] *** [2026-05-08T10:26:20.429Z] ok: [an-a01n02] [2026-05-08T10:26:20.429Z] [2026-05-08T10:26:20.429Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T10:26:20.429Z] ] *** [2026-05-08T10:26:21.170Z] ok: [an-a01n02] [2026-05-08T10:26:21.170Z] [2026-05-08T10:26:21.170Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-05-08T10:26:21.516Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-05-08T10:26:21.516Z] [2026-05-08T10:26:21.516Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n01 -> localhost] [2026-05-08T10:29:46.121Z] ok: [an-a01n02 -> localhost] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:29:46.121Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:29:46.121Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:29:46.121Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] PLAY [basic server tests] ****************************************************** [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:29:46.121Z] ok: [an-a01n01] [2026-05-08T10:29:46.121Z] ok: [an-a01n02] [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:46.121Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:29:46.121Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:29:46.121Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:29:46.121Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-08T10:29:46.121Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:46.121Z] [2026-05-08T10:29:46.121Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-08T10:29:46.121Z] if [ -z "$storage" ]; then [2026-05-08T10:29:46.121Z] echo "Unable to detect shared storage" [2026-05-08T10:29:46.121Z] exit 1 [2026-05-08T10:29:46.121Z] fi [2026-05-08T10:29:46.121Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:29:46.121Z] ] *** [2026-05-08T10:29:51.000Z] changed: [an-a01n01] [2026-05-08T10:29:51.000Z] changed: [an-a01n02] [2026-05-08T10:29:51.000Z] [2026-05-08T10:29:51.000Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-08T10:29:51.000Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:51.000Z] [2026-05-08T10:29:51.000Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-08T10:29:51.000Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-08T10:29:51.000Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-08T10:29:51.000Z] echo "Unable to detect all deploy iso" [2026-05-08T10:29:51.000Z] echo "Detected isos" [2026-05-08T10:29:51.000Z] echo "$isos" [2026-05-08T10:29:51.000Z] exit 1 [2026-05-08T10:29:51.000Z] fi [2026-05-08T10:29:51.000Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-08T10:29:51.000Z] if [ -z "$installiso" ]; then [2026-05-08T10:29:51.000Z] echo "Unable to detect install iso" [2026-05-08T10:29:51.000Z] echo "Detected isos" [2026-05-08T10:29:51.000Z] echo "$isos" [2026-05-08T10:29:51.000Z] exit 1 [2026-05-08T10:29:51.000Z] fi [2026-05-08T10:29:51.000Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:29:51.000Z] ] *** [2026-05-08T10:29:57.047Z] changed: [an-a01n02] [2026-05-08T10:29:57.047Z] changed: [an-a01n01] [2026-05-08T10:29:57.047Z] [2026-05-08T10:29:57.047Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-08T10:29:57.047Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:57.047Z] [2026-05-08T10:29:57.047Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:29:57.047Z] ] *** [2026-05-08T10:29:57.787Z] skipping: [an-a01n02] [2026-05-08T10:29:57.787Z] ok: [an-a01n01] [2026-05-08T10:29:57.787Z] [2026-05-08T10:29:57.787Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T10:29:57.787Z] --ci-test \ [2026-05-08T10:29:57.787Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T10:29:57.787Z] --os almalinux9 \ [2026-05-08T10:29:57.787Z] --cpu 2 --ram 4G \ [2026-05-08T10:29:57.787Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T10:29:57.787Z] --install-media {{ installisouuid }} \ [2026-05-08T10:29:57.787Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T10:29:57.787Z] ] *** [2026-05-08T10:29:57.787Z] skipping: [an-a01n01] [2026-05-08T10:29:57.787Z] skipping: [an-a01n02] [2026-05-08T10:29:57.787Z] [2026-05-08T10:29:57.787Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T10:29:57.787Z] skipping: [an-a01n01] [2026-05-08T10:29:57.787Z] skipping: [an-a01n02] [2026-05-08T10:29:57.787Z] [2026-05-08T10:29:57.787Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:29:57.787Z] skipping: [an-a01n01] [2026-05-08T10:29:57.787Z] skipping: [an-a01n02] [2026-05-08T10:29:57.787Z] [2026-05-08T10:29:57.787Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T10:29:57.787Z] skipping: [an-a01n01] [2026-05-08T10:29:57.787Z] skipping: [an-a01n02] [2026-05-08T10:29:57.787Z] [2026-05-08T10:29:57.787Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T10:29:58.132Z] skipping: [an-a01n01] [2026-05-08T10:29:58.132Z] skipping: [an-a01n02] [2026-05-08T10:29:58.132Z] [2026-05-08T10:29:58.132Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-05-08T10:29:58.132Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:58.132Z] [2026-05-08T10:29:58.132Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:29:58.132Z] ] *** [2026-05-08T10:29:59.536Z] skipping: [an-a01n01] [2026-05-08T10:29:59.536Z] ok: [an-a01n02] [2026-05-08T10:29:59.536Z] [2026-05-08T10:29:59.536Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T10:29:59.536Z] --ci-test \ [2026-05-08T10:29:59.536Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T10:29:59.536Z] --os almalinux9 \ [2026-05-08T10:29:59.536Z] --cpu 2 --ram 4G \ [2026-05-08T10:29:59.536Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T10:29:59.536Z] --install-media {{ installisouuid }} \ [2026-05-08T10:29:59.536Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T10:29:59.536Z] ] *** [2026-05-08T10:29:59.536Z] skipping: [an-a01n01] [2026-05-08T10:29:59.536Z] skipping: [an-a01n02] [2026-05-08T10:29:59.536Z] [2026-05-08T10:29:59.536Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T10:29:59.536Z] skipping: [an-a01n01] [2026-05-08T10:29:59.536Z] skipping: [an-a01n02] [2026-05-08T10:29:59.536Z] [2026-05-08T10:29:59.536Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:29:59.880Z] skipping: [an-a01n01] [2026-05-08T10:29:59.880Z] skipping: [an-a01n02] [2026-05-08T10:29:59.880Z] [2026-05-08T10:29:59.880Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T10:29:59.880Z] skipping: [an-a01n01] [2026-05-08T10:29:59.880Z] skipping: [an-a01n02] [2026-05-08T10:29:59.880Z] [2026-05-08T10:29:59.880Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T10:29:59.880Z] skipping: [an-a01n01] [2026-05-08T10:29:59.880Z] skipping: [an-a01n02] [2026-05-08T10:29:59.881Z] [2026-05-08T10:29:59.881Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-08T10:29:59.881Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:29:59.881Z] [2026-05-08T10:29:59.881Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:29:59.881Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:29:59.881Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:29:59.881Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:29:59.881Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:29:59.881Z] echo "pcs:" [2026-05-08T10:29:59.881Z] echo "$pcsstatus" [2026-05-08T10:29:59.881Z] exit 1 [2026-05-08T10:29:59.881Z] fi [2026-05-08T10:29:59.881Z] exit 0 [2026-05-08T10:29:59.881Z] ] *** [2026-05-08T10:30:02.215Z] ok: [an-a01n02] [2026-05-08T10:30:02.215Z] ok: [an-a01n01] [2026-05-08T10:30:02.215Z] [2026-05-08T10:30:02.215Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:30:02.215Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:30:02.215Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:30:02.215Z] ret=0 [2026-05-08T10:30:02.215Z] if [ "$servers" != "1" ]; then [2026-05-08T10:30:02.215Z] echo "Not all servers are running on the same node!" [2026-05-08T10:30:02.215Z] echo "$pcsstatus" [2026-05-08T10:30:02.215Z] ret=1 [2026-05-08T10:30:02.215Z] fi [2026-05-08T10:30:02.215Z] echo "$hypervisor" [2026-05-08T10:30:02.215Z] exit $ret [2026-05-08T10:30:02.215Z] ] *** [2026-05-08T10:30:03.937Z] ok: [an-a01n01] [2026-05-08T10:30:03.937Z] ok: [an-a01n02] [2026-05-08T10:30:03.937Z] [2026-05-08T10:30:03.937Z] PLAY [delete extra server tests] *********************************************** [2026-05-08T10:30:03.937Z] [2026-05-08T10:30:03.937Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:30:05.659Z] ok: [an-a01n01] [2026-05-08T10:30:05.659Z] ok: [an-a01n02] [2026-05-08T10:30:05.659Z] [2026-05-08T10:30:05.659Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T10:30:05.659Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:30:05.659Z] [2026-05-08T10:30:05.659Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:30:05.659Z] ] *** [2026-05-08T10:30:09.551Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:30:09.551Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:30:09.551Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:30:09.551Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:30:09.551Z] [2026-05-08T10:30:09.551Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-05-08T10:30:09.896Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T10:30:09.896Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T10:30:09.896Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T10:30:09.896Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T10:30:09.896Z] [2026-05-08T10:30:09.896Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:30:09.896Z] ] *** [2026-05-08T10:30:11.619Z] skipping: [an-a01n01] [2026-05-08T10:30:11.619Z] ok: [an-a01n02] [2026-05-08T10:30:11.619Z] [2026-05-08T10:30:11.619Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T10:30:11.619Z] --force \ [2026-05-08T10:30:11.619Z] --server an-test-deploy{{ servernum }} [2026-05-08T10:30:11.619Z] ] *** [2026-05-08T10:30:17.667Z] skipping: [an-a01n01] [2026-05-08T10:30:17.667Z] changed: [an-a01n02] [2026-05-08T10:30:17.667Z] [2026-05-08T10:30:17.667Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:30:17.667Z] ] *** [2026-05-08T10:30:45.861Z] skipping: [an-a01n01] [2026-05-08T10:30:45.861Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-05-08T10:30:45.861Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-05-08T10:30:45.861Z] ok: [an-a01n02] [2026-05-08T10:30:45.861Z] [2026-05-08T10:30:45.861Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T10:30:45.861Z] ] *** [2026-05-08T10:30:45.861Z] skipping: [an-a01n01] [2026-05-08T10:30:45.861Z] ok: [an-a01n02] [2026-05-08T10:30:45.861Z] [2026-05-08T10:30:45.861Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T10:30:45.861Z] ] *** [2026-05-08T10:30:45.861Z] skipping: [an-a01n01] [2026-05-08T10:30:45.861Z] ok: [an-a01n02] [2026-05-08T10:30:45.861Z] [2026-05-08T10:30:45.861Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:30:45.861Z] ] *** [2026-05-08T10:30:45.861Z] skipping: [an-a01n01] [2026-05-08T10:30:45.861Z] ok: [an-a01n02] [2026-05-08T10:30:45.861Z] [2026-05-08T10:30:45.861Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T10:30:45.861Z] --force \ [2026-05-08T10:30:45.861Z] --server an-test-deploy{{ servernum }} [2026-05-08T10:30:45.861Z] ] *** [2026-05-08T10:30:50.743Z] skipping: [an-a01n01] [2026-05-08T10:30:50.743Z] changed: [an-a01n02] [2026-05-08T10:30:50.743Z] [2026-05-08T10:30:50.743Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:30:50.743Z] ] *** [2026-05-08T10:32:00.560Z] skipping: [an-a01n01] [2026-05-08T10:32:00.560Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-05-08T10:32:00.560Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-05-08T10:32:00.560Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-05-08T10:32:00.560Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-05-08T10:32:00.560Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-05-08T10:32:00.560Z] ok: [an-a01n02] [2026-05-08T10:32:00.560Z] [2026-05-08T10:32:00.560Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T10:32:00.560Z] ] *** [2026-05-08T10:32:00.560Z] skipping: [an-a01n01] [2026-05-08T10:32:00.560Z] ok: [an-a01n02] [2026-05-08T10:32:00.560Z] [2026-05-08T10:32:00.560Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T10:32:00.560Z] ] *** [2026-05-08T10:32:00.560Z] skipping: [an-a01n01] [2026-05-08T10:32:00.560Z] ok: [an-a01n02] [2026-05-08T10:32:00.560Z] [2026-05-08T10:32:00.560Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:32:00.560Z] ] *** [2026-05-08T10:32:00.560Z] skipping: [an-a01n01] [2026-05-08T10:32:00.560Z] ok: [an-a01n02] [2026-05-08T10:32:00.560Z] [2026-05-08T10:32:00.560Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T10:32:00.560Z] --force \ [2026-05-08T10:32:00.560Z] --server an-test-deploy{{ servernum }} [2026-05-08T10:32:00.560Z] ] *** [2026-05-08T10:32:00.560Z] skipping: [an-a01n01] [2026-05-08T10:32:00.560Z] changed: [an-a01n02] [2026-05-08T10:32:00.560Z] [2026-05-08T10:32:00.560Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:32:00.560Z] ] *** [2026-05-08T10:32:58.606Z] skipping: [an-a01n01] [2026-05-08T10:32:58.606Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-05-08T10:32:58.606Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-05-08T10:32:58.606Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-05-08T10:32:58.606Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-05-08T10:32:58.606Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-05-08T10:32:58.606Z] ok: [an-a01n02] [2026-05-08T10:32:58.606Z] [2026-05-08T10:32:58.606Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T10:32:58.606Z] ] *** [2026-05-08T10:32:58.606Z] skipping: [an-a01n01] [2026-05-08T10:32:58.606Z] ok: [an-a01n02] [2026-05-08T10:32:58.606Z] [2026-05-08T10:32:58.606Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T10:32:58.606Z] ] *** [2026-05-08T10:32:58.951Z] skipping: [an-a01n01] [2026-05-08T10:32:58.951Z] ok: [an-a01n02] [2026-05-08T10:32:58.951Z] [2026-05-08T10:32:58.951Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:32:58.951Z] ] *** [2026-05-08T10:33:00.832Z] skipping: [an-a01n01] [2026-05-08T10:33:00.832Z] ok: [an-a01n02] [2026-05-08T10:33:00.832Z] [2026-05-08T10:33:00.832Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T10:33:00.832Z] --force \ [2026-05-08T10:33:00.832Z] --server an-test-deploy{{ servernum }} [2026-05-08T10:33:00.832Z] ] *** [2026-05-08T10:33:06.886Z] skipping: [an-a01n01] [2026-05-08T10:33:06.886Z] changed: [an-a01n02] [2026-05-08T10:33:06.886Z] [2026-05-08T10:33:06.886Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:33:06.886Z] ] *** [2026-05-08T10:34:16.500Z] skipping: [an-a01n01] [2026-05-08T10:34:16.500Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-05-08T10:34:16.500Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-05-08T10:34:16.500Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-05-08T10:34:16.500Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-05-08T10:34:16.500Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-05-08T10:34:16.500Z] ok: [an-a01n02] [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T10:34:16.500Z] ] *** [2026-05-08T10:34:16.500Z] skipping: [an-a01n01] [2026-05-08T10:34:16.500Z] ok: [an-a01n02] [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T10:34:16.500Z] ] *** [2026-05-08T10:34:16.500Z] skipping: [an-a01n01] [2026-05-08T10:34:16.500Z] ok: [an-a01n02] [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] PLAY [create server for DR testing] ******************************************** [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:34:16.500Z] ok: [an-a01n02] [2026-05-08T10:34:16.500Z] ok: [an-a01n01] [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T10:34:16.500Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:34:16.500Z] ] *** [2026-05-08T10:34:16.500Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:34:16.500Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:34:16.500Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:34:16.500Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-08T10:34:16.500Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:16.500Z] [2026-05-08T10:34:16.500Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-08T10:34:16.500Z] if [ -z "$storage" ]; then [2026-05-08T10:34:16.500Z] echo "Unable to detect shared storage" [2026-05-08T10:34:16.500Z] exit 1 [2026-05-08T10:34:16.500Z] fi [2026-05-08T10:34:16.500Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:34:16.500Z] ] *** [2026-05-08T10:34:18.360Z] changed: [an-a01n01] [2026-05-08T10:34:18.360Z] changed: [an-a01n02] [2026-05-08T10:34:18.360Z] [2026-05-08T10:34:18.360Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-08T10:34:18.705Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:18.705Z] [2026-05-08T10:34:18.705Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-08T10:34:18.705Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-08T10:34:18.705Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-08T10:34:18.705Z] echo "Unable to detect all deploy iso" [2026-05-08T10:34:18.705Z] echo "Detected isos" [2026-05-08T10:34:18.705Z] echo "$isos" [2026-05-08T10:34:18.705Z] exit 1 [2026-05-08T10:34:18.705Z] fi [2026-05-08T10:34:18.705Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-08T10:34:18.705Z] if [ -z "$installiso" ]; then [2026-05-08T10:34:18.705Z] echo "Unable to detect install iso" [2026-05-08T10:34:18.705Z] echo "Detected isos" [2026-05-08T10:34:18.705Z] echo "$isos" [2026-05-08T10:34:18.705Z] exit 1 [2026-05-08T10:34:18.705Z] fi [2026-05-08T10:34:18.705Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:34:18.705Z] ] *** [2026-05-08T10:34:24.754Z] changed: [an-a01n02] [2026-05-08T10:34:24.754Z] changed: [an-a01n01] [2026-05-08T10:34:24.754Z] [2026-05-08T10:34:24.754Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-08T10:34:24.754Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:24.754Z] [2026-05-08T10:34:24.754Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:34:24.754Z] ] *** [2026-05-08T10:34:25.494Z] skipping: [an-a01n02] [2026-05-08T10:34:25.494Z] ok: [an-a01n01] [2026-05-08T10:34:25.494Z] [2026-05-08T10:34:25.494Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T10:34:25.494Z] --ci-test \ [2026-05-08T10:34:25.494Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T10:34:25.494Z] --os almalinux9 \ [2026-05-08T10:34:25.494Z] --cpu 2 --ram 4G \ [2026-05-08T10:34:25.494Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T10:34:25.494Z] --install-media {{ installisouuid }} \ [2026-05-08T10:34:25.494Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T10:34:25.494Z] ] *** [2026-05-08T10:34:25.494Z] skipping: [an-a01n01] [2026-05-08T10:34:25.494Z] skipping: [an-a01n02] [2026-05-08T10:34:25.494Z] [2026-05-08T10:34:25.494Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T10:34:25.494Z] skipping: [an-a01n01] [2026-05-08T10:34:25.494Z] skipping: [an-a01n02] [2026-05-08T10:34:25.494Z] [2026-05-08T10:34:25.494Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:34:25.494Z] skipping: [an-a01n01] [2026-05-08T10:34:25.494Z] skipping: [an-a01n02] [2026-05-08T10:34:25.494Z] [2026-05-08T10:34:25.494Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T10:34:25.838Z] skipping: [an-a01n01] [2026-05-08T10:34:25.838Z] skipping: [an-a01n02] [2026-05-08T10:34:25.838Z] [2026-05-08T10:34:25.838Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T10:34:25.838Z] skipping: [an-a01n01] [2026-05-08T10:34:25.838Z] skipping: [an-a01n02] [2026-05-08T10:34:25.838Z] [2026-05-08T10:34:25.838Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-08T10:34:25.838Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:25.839Z] [2026-05-08T10:34:25.839Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:34:25.839Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:34:25.839Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:34:25.839Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:34:25.839Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:34:25.839Z] echo "pcs:" [2026-05-08T10:34:25.839Z] echo "$pcsstatus" [2026-05-08T10:34:25.839Z] exit 1 [2026-05-08T10:34:25.839Z] fi [2026-05-08T10:34:25.839Z] exit 0 [2026-05-08T10:34:25.839Z] ] *** [2026-05-08T10:34:28.174Z] ok: [an-a01n02] [2026-05-08T10:34:28.174Z] ok: [an-a01n01] [2026-05-08T10:34:28.174Z] [2026-05-08T10:34:28.174Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:34:28.174Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:34:28.174Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:34:28.174Z] ret=0 [2026-05-08T10:34:28.174Z] if [ "$servers" != "1" ]; then [2026-05-08T10:34:28.174Z] echo "Not all servers are running on the same node!" [2026-05-08T10:34:28.174Z] echo "$pcsstatus" [2026-05-08T10:34:28.174Z] ret=1 [2026-05-08T10:34:28.174Z] fi [2026-05-08T10:34:28.174Z] echo "$hypervisor" [2026-05-08T10:34:28.174Z] exit $ret [2026-05-08T10:34:28.174Z] ] *** [2026-05-08T10:34:29.896Z] ok: [an-a01n02] [2026-05-08T10:34:29.896Z] ok: [an-a01n01] [2026-05-08T10:34:29.896Z] [2026-05-08T10:34:29.896Z] PLAY [DR tests] **************************************************************** [2026-05-08T10:34:29.896Z] skipping: no hosts matched [2026-05-08T10:34:29.896Z] [2026-05-08T10:34:29.896Z] PLAY [upgrade server test] ***************************************************** [2026-05-08T10:34:29.896Z] [2026-05-08T10:34:29.896Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:34:31.623Z] ok: [an-a01n01] [2026-05-08T10:34:31.623Z] ok: [an-a01n02] [2026-05-08T10:34:31.623Z] [2026-05-08T10:34:31.623Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T10:34:31.975Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:31.975Z] [2026-05-08T10:34:31.975Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T10:34:31.975Z] ] *** [2026-05-08T10:34:36.007Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T10:34:36.007Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T10:34:36.007Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T10:34:36.007Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T10:34:36.007Z] [2026-05-08T10:34:36.007Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-05-08T10:34:36.007Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:36.007Z] [2026-05-08T10:34:36.007Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-05-08T10:34:36.007Z] if [ -z "$storage" ]; then [2026-05-08T10:34:36.007Z] echo "Unable to detect shared storage" [2026-05-08T10:34:36.007Z] exit 1 [2026-05-08T10:34:36.007Z] fi [2026-05-08T10:34:36.007Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:34:36.007Z] ] *** [2026-05-08T10:34:40.887Z] changed: [an-a01n02] [2026-05-08T10:34:40.887Z] changed: [an-a01n01] [2026-05-08T10:34:40.887Z] [2026-05-08T10:34:40.887Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-05-08T10:34:40.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:40.887Z] [2026-05-08T10:34:40.887Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-05-08T10:34:40.887Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-05-08T10:34:40.887Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-05-08T10:34:40.887Z] echo "Unable to detect all deploy iso" [2026-05-08T10:34:40.887Z] echo "Detected isos" [2026-05-08T10:34:40.887Z] echo "$isos" [2026-05-08T10:34:40.887Z] exit 1 [2026-05-08T10:34:40.887Z] fi [2026-05-08T10:34:40.887Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-05-08T10:34:40.887Z] if [ -z "$installiso" ]; then [2026-05-08T10:34:40.887Z] echo "Unable to detect install iso" [2026-05-08T10:34:40.887Z] echo "Detected isos" [2026-05-08T10:34:40.887Z] echo "$isos" [2026-05-08T10:34:40.887Z] exit 1 [2026-05-08T10:34:40.887Z] fi [2026-05-08T10:34:40.887Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-05-08T10:34:40.887Z] ] *** [2026-05-08T10:34:46.935Z] changed: [an-a01n02] [2026-05-08T10:34:46.935Z] changed: [an-a01n01] [2026-05-08T10:34:46.935Z] [2026-05-08T10:34:46.935Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-05-08T10:34:46.935Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:46.935Z] [2026-05-08T10:34:46.935Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T10:34:46.935Z] ] *** [2026-05-08T10:34:47.676Z] skipping: [an-a01n02] [2026-05-08T10:34:47.676Z] ok: [an-a01n01] [2026-05-08T10:34:47.676Z] [2026-05-08T10:34:47.676Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage c2309e5f-241c-4dbe-a5b4-5a161442d2b0 _raw_params=anvil-provision-server -vv --log-secure \ [2026-05-08T10:34:47.676Z] --ci-test \ [2026-05-08T10:34:47.676Z] --name an-test-deploy{{ servernum }} \ [2026-05-08T10:34:47.676Z] --os almalinux9 \ [2026-05-08T10:34:47.676Z] --cpu 2 --ram 4G \ [2026-05-08T10:34:47.676Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-05-08T10:34:47.676Z] --install-media {{ installisouuid }} \ [2026-05-08T10:34:47.676Z] --driver-disc deploy{{ servernum }}.iso [2026-05-08T10:34:47.676Z] ] *** [2026-05-08T10:34:47.676Z] skipping: [an-a01n01] [2026-05-08T10:34:47.676Z] skipping: [an-a01n02] [2026-05-08T10:34:47.676Z] [2026-05-08T10:34:47.676Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-05-08T10:34:47.676Z] skipping: [an-a01n01] [2026-05-08T10:34:47.676Z] skipping: [an-a01n02] [2026-05-08T10:34:47.676Z] [2026-05-08T10:34:47.676Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-05-08T10:34:48.021Z] skipping: [an-a01n01] [2026-05-08T10:34:48.021Z] skipping: [an-a01n02] [2026-05-08T10:34:48.021Z] [2026-05-08T10:34:48.021Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-05-08T10:34:48.021Z] skipping: [an-a01n01] [2026-05-08T10:34:48.021Z] skipping: [an-a01n02] [2026-05-08T10:34:48.021Z] [2026-05-08T10:34:48.021Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-05-08T10:34:48.021Z] skipping: [an-a01n01] [2026-05-08T10:34:48.021Z] skipping: [an-a01n02] [2026-05-08T10:34:48.021Z] [2026-05-08T10:34:48.021Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-05-08T10:34:48.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-05-08T10:34:48.021Z] [2026-05-08T10:34:48.021Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:34:48.021Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-05-08T10:34:48.021Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-05-08T10:34:48.021Z] if [ "$numservers" != "$startedservers" ]; then [2026-05-08T10:34:48.021Z] echo "Detected $numservers but only $startedservers in Started state" [2026-05-08T10:34:48.021Z] echo "pcs:" [2026-05-08T10:34:48.021Z] echo "$pcsstatus" [2026-05-08T10:34:48.021Z] exit 1 [2026-05-08T10:34:48.021Z] fi [2026-05-08T10:34:48.021Z] exit 0 [2026-05-08T10:34:48.021Z] ] *** [2026-05-08T10:34:50.355Z] ok: [an-a01n01] [2026-05-08T10:34:50.355Z] ok: [an-a01n02] [2026-05-08T10:34:50.355Z] [2026-05-08T10:34:50.355Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-05-08T10:34:50.355Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-05-08T10:34:50.355Z] servers="$(echo "$hypervisor" | wc -l)" [2026-05-08T10:34:50.355Z] ret=0 [2026-05-08T10:34:50.355Z] if [ "$servers" != "1" ]; then [2026-05-08T10:34:50.355Z] echo "Not all servers are running on the same node!" [2026-05-08T10:34:50.355Z] echo "$pcsstatus" [2026-05-08T10:34:50.355Z] ret=1 [2026-05-08T10:34:50.355Z] fi [2026-05-08T10:34:50.355Z] echo "$hypervisor" [2026-05-08T10:34:50.355Z] exit $ret [2026-05-08T10:34:50.355Z] ] *** [2026-05-08T10:34:52.078Z] ok: [an-a01n02] [2026-05-08T10:34:52.078Z] ok: [an-a01n01] [2026-05-08T10:34:52.078Z] [2026-05-08T10:34:52.078Z] PLAY [enable upgrade repo] ***************************************************** [2026-05-08T10:34:52.078Z] [2026-05-08T10:34:52.078Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:34:54.413Z] ok: [an-a01n02] [2026-05-08T10:34:54.413Z] ok: [an-a01n01] [2026-05-08T10:34:54.413Z] ok: [an-striker02] [2026-05-08T10:34:54.413Z] ok: [an-striker01] [2026-05-08T10:34:54.413Z] [2026-05-08T10:34:54.413Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-05-08T10:34:54.413Z] dnf config-manager --set-enabled $updaterepo [2026-05-08T10:34:54.413Z] ] *** [2026-05-08T10:35:05.566Z] changed: [an-a01n02] [2026-05-08T10:35:05.566Z] changed: [an-a01n01] [2026-05-08T10:35:05.566Z] changed: [an-striker02] [2026-05-08T10:35:05.566Z] changed: [an-striker01] [2026-05-08T10:35:05.566Z] [2026-05-08T10:35:05.566Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-05-08T10:35:05.566Z] ok: [an-striker01] [2026-05-08T10:35:05.566Z] ok: [an-striker02] [2026-05-08T10:35:05.566Z] ok: [an-a01n01] [2026-05-08T10:35:05.566Z] ok: [an-a01n02] [2026-05-08T10:35:05.566Z] [2026-05-08T10:35:05.566Z] PLAY [start upgrade process] *************************************************** [2026-05-08T10:35:05.566Z] [2026-05-08T10:35:05.566Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:35:07.286Z] ok: [an-striker01] [2026-05-08T10:35:07.286Z] [2026-05-08T10:35:07.286Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-05-08T10:35:07.286Z] ] *** [2026-05-08T10:56:53.730Z] changed: [an-striker01] [2026-05-08T10:56:53.730Z] [2026-05-08T10:56:53.730Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-05-08T10:57:17.042Z] changed: [an-striker01] [2026-05-08T10:57:17.042Z] [2026-05-08T10:57:17.042Z] PLAY [reconnect all nodes] ***************************************************** [2026-05-08T10:57:17.042Z] [2026-05-08T10:57:17.042Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T10:57:21.266Z] ok: [an-a01n02] [2026-05-08T10:57:21.266Z] ok: [an-a01n01] [2026-05-08T10:57:21.266Z] ok: [an-striker02] [2026-05-08T10:57:21.266Z] ok: [an-striker01] [2026-05-08T10:57:21.266Z] [2026-05-08T10:57:21.266Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-05-08T10:57:21.266Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02 [2026-05-08T10:57:21.266Z] [2026-05-08T10:57:21.266Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-05-08T11:00:45.830Z] ok: [an-a01n02 -> localhost] [2026-05-08T11:00:45.830Z] ok: [an-striker01 -> localhost] [2026-05-08T11:00:45.830Z] ok: [an-striker02 -> localhost] [2026-05-08T11:00:45.830Z] ok: [an-a01n01 -> localhost] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-05-08T11:00:45.830Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-05-08T11:00:45.830Z] exit 1 [2026-05-08T11:00:45.830Z] fi [2026-05-08T11:00:45.830Z] ] *** [2026-05-08T11:00:45.830Z] fatal: [an-a01n01]: FAILED! => {"changed": true, "cmd": "if [ \"649\" -ge \"599\" ]; then\n echo \"Node current uptime higher than expected! tool failed to reboot nodes?\"\n exit 1\nfi\n", "delta": "0:00:00.004836", "end": "2026-05-08 07:00:32.102087", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 07:00:32.097251", "stderr": "", "stderr_lines": [], "stdout": "Node current uptime higher than expected! tool failed to reboot nodes?", "stdout_lines": ["Node current uptime higher than expected! tool failed to reboot nodes?"]} [2026-05-08T11:00:45.830Z] ...ignoring [2026-05-08T11:00:45.830Z] changed: [an-a01n02] [2026-05-08T11:00:45.830Z] changed: [an-striker01] [2026-05-08T11:00:45.830Z] changed: [an-striker02] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] PLAY [check all nodes status after upgrade] ************************************ [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:00:45.830Z] ok: [an-a01n02] [2026-05-08T11:00:45.830Z] ok: [an-striker01] [2026-05-08T11:00:45.830Z] ok: [an-striker02] [2026-05-08T11:00:45.830Z] ok: [an-a01n01] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-05-08T11:00:45.830Z] echo "perl-Log-Journald update not found!" [2026-05-08T11:00:45.830Z] exit 1 [2026-05-08T11:00:45.830Z] fi [2026-05-08T11:00:45.830Z] ] *** [2026-05-08T11:00:45.830Z] changed: [an-a01n01] [2026-05-08T11:00:45.830Z] changed: [an-a01n02] [2026-05-08T11:00:45.830Z] changed: [an-striker01] [2026-05-08T11:00:45.830Z] changed: [an-striker02] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] PLAY [check nodes status after upgrade] **************************************** [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:00:45.830Z] ok: [an-a01n01] [2026-05-08T11:00:45.830Z] ok: [an-a01n02] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-05-08T11:00:45.830Z] echo "akmod-drbd update not found!" [2026-05-08T11:00:45.830Z] exit 1 [2026-05-08T11:00:45.830Z] fi [2026-05-08T11:00:45.830Z] ] *** [2026-05-08T11:00:45.830Z] changed: [an-a01n02] [2026-05-08T11:00:45.830Z] changed: [an-a01n01] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] PLAY [check hosted node status after upgrade] ********************************** [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:00:45.830Z] ok: [an-a01n01] [2026-05-08T11:00:45.830Z] ok: [an-a01n02] [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T11:00:45.830Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T11:00:45.830Z] [2026-05-08T11:00:45.830Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T11:00:45.830Z] ] *** [2026-05-08T11:00:46.175Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T11:00:46.175Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T11:00:46.175Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T11:00:46.175Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T11:00:46.175Z] [2026-05-08T11:00:46.175Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-05-08T11:00:46.175Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-05-08T11:00:46.175Z] [2026-05-08T11:00:46.175Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-05-08T11:00:46.175Z] ] *** [2026-05-08T11:00:47.897Z] ok: [an-a01n01] [2026-05-08T11:00:47.897Z] ok: [an-a01n02] [2026-05-08T11:00:47.897Z] [2026-05-08T11:00:47.897Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-05-08T11:00:47.897Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-05-08T11:00:47.897Z] [2026-05-08T11:00:47.897Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-05-08T11:00:47.897Z] ] *** [2026-05-08T11:00:48.637Z] ok: [an-a01n01] [2026-05-08T11:00:48.637Z] ok: [an-a01n02] [2026-05-08T11:00:48.637Z] [2026-05-08T11:00:48.637Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-05-08T11:00:48.637Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-05-08T11:00:48.637Z] [2026-05-08T11:00:48.637Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-05-08T11:00:48.637Z] ] *** [2026-05-08T11:00:49.377Z] ok: [an-a01n01] [2026-05-08T11:00:49.377Z] ok: [an-a01n02] [2026-05-08T11:00:49.377Z] [2026-05-08T11:00:49.377Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-05-08T11:00:49.377Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-05-08T11:00:49.377Z] [2026-05-08T11:00:49.377Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-05-08T11:00:49.377Z] ] *** [2026-05-08T11:00:50.117Z] ok: [an-a01n01] [2026-05-08T11:00:50.117Z] ok: [an-a01n02] [2026-05-08T11:00:50.117Z] [2026-05-08T11:00:50.117Z] PLAY [test anvil package self update] ****************************************** [2026-05-08T11:00:50.117Z] [2026-05-08T11:00:50.117Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:00:52.453Z] ok: [an-striker01] [2026-05-08T11:00:52.453Z] ok: [an-a01n02] [2026-05-08T11:00:52.453Z] ok: [an-a01n01] [2026-05-08T11:00:52.453Z] ok: [an-striker02] [2026-05-08T11:00:52.453Z] [2026-05-08T11:00:52.453Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-05-08T11:00:55.505Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-05-08T11:00:55.505Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-05-08T11:00:55.505Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-05-08T11:00:55.505Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-05-08T11:00:55.505Z] changed: [an-a01n02] => (item=scancore.service) [2026-05-08T11:00:55.505Z] changed: [an-a01n01] => (item=scancore.service) [2026-05-08T11:00:55.505Z] changed: [an-striker01] => (item=scancore.service) [2026-05-08T11:00:55.505Z] changed: [an-striker02] => (item=scancore.service) [2026-05-08T11:00:55.505Z] [2026-05-08T11:00:55.505Z] PLAY [test striker package self update] **************************************** [2026-05-08T11:00:55.505Z] [2026-05-08T11:00:55.505Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:00:57.225Z] ok: [an-striker01] [2026-05-08T11:00:57.225Z] ok: [an-striker02] [2026-05-08T11:00:57.225Z] [2026-05-08T11:00:57.225Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-05-08T11:00:57.225Z] ] *** [2026-05-08T11:01:25.452Z] changed: [an-striker01] [2026-05-08T11:01:25.452Z] changed: [an-striker02] [2026-05-08T11:01:25.452Z] [2026-05-08T11:01:25.452Z] PLAY [test node package self update] ******************************************* [2026-05-08T11:01:25.452Z] [2026-05-08T11:01:25.452Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:01:25.452Z] ok: [an-a01n02] [2026-05-08T11:01:25.452Z] ok: [an-a01n01] [2026-05-08T11:01:25.452Z] [2026-05-08T11:01:25.452Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-05-08T11:01:25.452Z] ] *** [2026-05-08T11:01:44.834Z] changed: [an-a01n02] [2026-05-08T11:01:44.834Z] changed: [an-a01n01] [2026-05-08T11:01:44.834Z] [2026-05-08T11:01:44.834Z] PLAY [test dr package self update] ********************************************* [2026-05-08T11:01:44.834Z] skipping: no hosts matched [2026-05-08T11:01:44.834Z] [2026-05-08T11:01:44.834Z] PLAY [test anvil package self update] ****************************************** [2026-05-08T11:01:44.834Z] [2026-05-08T11:01:44.834Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:01:46.556Z] ok: [an-striker02] [2026-05-08T11:01:46.556Z] ok: [an-striker01] [2026-05-08T11:01:46.556Z] ok: [an-a01n02] [2026-05-08T11:01:46.556Z] ok: [an-a01n01] [2026-05-08T11:01:46.556Z] [2026-05-08T11:01:46.556Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-05-08T11:01:49.607Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-05-08T11:01:49.607Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-05-08T11:01:49.607Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-05-08T11:01:49.607Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-05-08T11:01:49.607Z] changed: [an-a01n02] => (item=scancore.service) [2026-05-08T11:01:49.607Z] changed: [an-a01n01] => (item=scancore.service) [2026-05-08T11:01:49.607Z] changed: [an-striker01] => (item=scancore.service) [2026-05-08T11:01:49.607Z] changed: [an-striker02] => (item=scancore.service) [2026-05-08T11:01:49.607Z] [2026-05-08T11:01:49.607Z] PLAY [delete server tests] ***************************************************** [2026-05-08T11:01:49.607Z] [2026-05-08T11:01:49.607Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:01:50.986Z] ok: [an-a01n02] [2026-05-08T11:01:50.986Z] ok: [an-a01n01] [2026-05-08T11:01:50.986Z] [2026-05-08T11:01:50.986Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-05-08T11:01:51.331Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-05-08T11:01:51.331Z] [2026-05-08T11:01:51.331Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-05-08T11:01:51.331Z] ] *** [2026-05-08T11:01:55.223Z] ok: [an-a01n02] => (item=an-a01n01) [2026-05-08T11:01:55.223Z] ok: [an-a01n01] => (item=an-a01n01) [2026-05-08T11:01:55.223Z] ok: [an-a01n02] => (item=an-a01n02) [2026-05-08T11:01:55.223Z] ok: [an-a01n01] => (item=an-a01n02) [2026-05-08T11:01:55.223Z] [2026-05-08T11:01:55.223Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-05-08T11:01:55.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-05-08T11:01:55.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-05-08T11:01:55.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-05-08T11:01:55.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-05-08T11:01:55.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-05-08T11:01:55.568Z] [2026-05-08T11:01:55.568Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:01:55.568Z] ] *** [2026-05-08T11:01:56.762Z] skipping: [an-a01n01] [2026-05-08T11:01:56.762Z] ok: [an-a01n02] [2026-05-08T11:01:56.762Z] [2026-05-08T11:01:56.762Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T11:01:56.762Z] --force \ [2026-05-08T11:01:56.762Z] --server an-test-deploy{{ servernum }} [2026-05-08T11:01:56.762Z] ] *** [2026-05-08T11:02:02.809Z] skipping: [an-a01n01] [2026-05-08T11:02:02.809Z] changed: [an-a01n02] [2026-05-08T11:02:02.809Z] [2026-05-08T11:02:02.809Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:02.809Z] ] *** [2026-05-08T11:02:51.435Z] skipping: [an-a01n01] [2026-05-08T11:02:51.435Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-05-08T11:02:51.435Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-05-08T11:02:51.435Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-05-08T11:02:51.435Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-05-08T11:02:51.435Z] ok: [an-a01n02] [2026-05-08T11:02:51.435Z] [2026-05-08T11:02:51.435Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T11:02:51.435Z] ] *** [2026-05-08T11:02:51.435Z] skipping: [an-a01n01] [2026-05-08T11:02:51.435Z] ok: [an-a01n02] [2026-05-08T11:02:51.435Z] [2026-05-08T11:02:51.435Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T11:02:51.435Z] ] *** [2026-05-08T11:02:51.435Z] skipping: [an-a01n01] [2026-05-08T11:02:51.435Z] ok: [an-a01n02] [2026-05-08T11:02:51.435Z] [2026-05-08T11:02:51.435Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:51.435Z] ] *** [2026-05-08T11:02:53.167Z] skipping: [an-a01n01] [2026-05-08T11:02:53.167Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.874779", "end": "2026-05-08 07:02:52.719607", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 07:02:51.844828", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T11:02:53.167Z] ...ignoring [2026-05-08T11:02:53.167Z] [2026-05-08T11:02:53.167Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T11:02:53.167Z] --force \ [2026-05-08T11:02:53.167Z] --server an-test-deploy{{ servernum }} [2026-05-08T11:02:53.167Z] ] *** [2026-05-08T11:02:53.167Z] skipping: [an-a01n01] [2026-05-08T11:02:53.167Z] skipping: [an-a01n02] [2026-05-08T11:02:53.167Z] [2026-05-08T11:02:53.167Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:53.167Z] ] *** [2026-05-08T11:02:53.167Z] skipping: [an-a01n01] [2026-05-08T11:02:53.167Z] skipping: [an-a01n02] [2026-05-08T11:02:53.167Z] [2026-05-08T11:02:53.167Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T11:02:53.167Z] ] *** [2026-05-08T11:02:53.167Z] skipping: [an-a01n01] [2026-05-08T11:02:53.167Z] skipping: [an-a01n02] [2026-05-08T11:02:53.167Z] [2026-05-08T11:02:53.167Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T11:02:53.167Z] ] *** [2026-05-08T11:02:53.167Z] skipping: [an-a01n01] [2026-05-08T11:02:53.167Z] skipping: [an-a01n02] [2026-05-08T11:02:53.167Z] [2026-05-08T11:02:53.167Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:53.167Z] ] *** [2026-05-08T11:02:54.892Z] skipping: [an-a01n01] [2026-05-08T11:02:54.892Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.924069", "end": "2026-05-08 07:02:54.477525", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 07:02:53.553456", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T11:02:54.892Z] ...ignoring [2026-05-08T11:02:54.892Z] [2026-05-08T11:02:54.892Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T11:02:54.892Z] --force \ [2026-05-08T11:02:54.892Z] --server an-test-deploy{{ servernum }} [2026-05-08T11:02:54.892Z] ] *** [2026-05-08T11:02:54.892Z] skipping: [an-a01n01] [2026-05-08T11:02:54.892Z] skipping: [an-a01n02] [2026-05-08T11:02:54.892Z] [2026-05-08T11:02:54.892Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:54.892Z] ] *** [2026-05-08T11:02:54.892Z] skipping: [an-a01n01] [2026-05-08T11:02:54.892Z] skipping: [an-a01n02] [2026-05-08T11:02:54.892Z] [2026-05-08T11:02:54.892Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T11:02:54.892Z] ] *** [2026-05-08T11:02:54.892Z] skipping: [an-a01n01] [2026-05-08T11:02:54.892Z] skipping: [an-a01n02] [2026-05-08T11:02:54.892Z] [2026-05-08T11:02:54.892Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T11:02:54.892Z] ] *** [2026-05-08T11:02:54.892Z] skipping: [an-a01n01] [2026-05-08T11:02:54.892Z] skipping: [an-a01n02] [2026-05-08T11:02:54.892Z] [2026-05-08T11:02:54.892Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:54.892Z] ] *** [2026-05-08T11:02:56.618Z] skipping: [an-a01n01] [2026-05-08T11:02:56.618Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.935843", "end": "2026-05-08 07:02:56.293616", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 07:02:55.357773", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T11:02:56.618Z] ...ignoring [2026-05-08T11:02:56.618Z] [2026-05-08T11:02:56.618Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T11:02:56.618Z] --force \ [2026-05-08T11:02:56.618Z] --server an-test-deploy{{ servernum }} [2026-05-08T11:02:56.618Z] ] *** [2026-05-08T11:02:56.618Z] skipping: [an-a01n01] [2026-05-08T11:02:56.618Z] skipping: [an-a01n02] [2026-05-08T11:02:56.618Z] [2026-05-08T11:02:56.618Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:56.618Z] ] *** [2026-05-08T11:02:56.618Z] skipping: [an-a01n01] [2026-05-08T11:02:56.618Z] skipping: [an-a01n02] [2026-05-08T11:02:56.618Z] [2026-05-08T11:02:56.618Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T11:02:56.618Z] ] *** [2026-05-08T11:02:56.618Z] skipping: [an-a01n01] [2026-05-08T11:02:56.618Z] skipping: [an-a01n02] [2026-05-08T11:02:56.618Z] [2026-05-08T11:02:56.618Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T11:02:56.618Z] ] *** [2026-05-08T11:02:56.964Z] skipping: [an-a01n01] [2026-05-08T11:02:56.964Z] skipping: [an-a01n02] [2026-05-08T11:02:56.964Z] [2026-05-08T11:02:56.964Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:56.964Z] ] *** [2026-05-08T11:02:58.162Z] skipping: [an-a01n01] [2026-05-08T11:02:58.162Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.870005", "end": "2026-05-08 07:02:58.052334", "msg": "non-zero return code", "rc": 1, "start": "2026-05-08 07:02:57.182329", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-05-08T11:02:58.162Z] ...ignoring [2026-05-08T11:02:58.162Z] [2026-05-08T11:02:58.162Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-05-08T11:02:58.162Z] --force \ [2026-05-08T11:02:58.162Z] --server an-test-deploy{{ servernum }} [2026-05-08T11:02:58.162Z] ] *** [2026-05-08T11:02:58.507Z] skipping: [an-a01n01] [2026-05-08T11:02:58.507Z] skipping: [an-a01n02] [2026-05-08T11:02:58.507Z] [2026-05-08T11:02:58.507Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-05-08T11:02:58.507Z] ] *** [2026-05-08T11:02:58.507Z] skipping: [an-a01n01] [2026-05-08T11:02:58.507Z] skipping: [an-a01n02] [2026-05-08T11:02:58.507Z] [2026-05-08T11:02:58.507Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-05-08T11:02:58.507Z] ] *** [2026-05-08T11:02:58.507Z] skipping: [an-a01n01] [2026-05-08T11:02:58.507Z] skipping: [an-a01n02] [2026-05-08T11:02:58.507Z] [2026-05-08T11:02:58.507Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-05-08T11:02:58.507Z] ] *** [2026-05-08T11:02:58.853Z] skipping: [an-a01n01] [2026-05-08T11:02:58.853Z] skipping: [an-a01n02] [2026-05-08T11:02:58.853Z] [2026-05-08T11:02:58.853Z] PLAY [Check anvil available resources from striker01] ************************** [2026-05-08T11:02:58.853Z] [2026-05-08T11:02:58.853Z] TASK [Gathering Facts ] ******************************************************** [2026-05-08T11:03:01.190Z] ok: [an-striker01] [2026-05-08T11:03:01.190Z] [2026-05-08T11:03:01.190Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-05-08T11:03:01.190Z] sleep 300 [2026-05-08T11:03:01.190Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-05-08T11:03:01.190Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-05-08T11:03:01.190Z] ] *** [2026-05-08T11:08:11.237Z] ok: [an-striker01] [2026-05-08T11:08:11.237Z] [2026-05-08T11:08:11.237Z] PLAY RECAP ********************************************************************* [2026-05-08T11:08:11.237Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=6 [2026-05-08T11:08:11.237Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-05-08T11:08:11.237Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-05-08T11:08:11.237Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-05-08T11:08:11.237Z] [2026-05-08T11:08:11.237Z] tar: Removing leading `/' from member names [2026-05-08T11:08:11.237Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-05-08T11:08:11.237Z] /root/ci-tools [2026-05-08T11:08:11.237Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/486.tar.gz [2026-05-08T11:08:11.237Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-05-08T11:08:11.237Z] This might take a few seconds. [2026-05-08T11:08:11.237Z] [2026-05-08T11:08:11.237Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-08T11:08:12.571Z] 𐄂 Disconnected from Red Hat Insights [2026-05-08T11:08:13.766Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-08T11:08:13.766Z] [2026-05-08T11:08:13.766Z] Manage your connected systems: https://red.ht/connector [2026-05-08T11:08:18.649Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-08T11:08:18.649Z] [2026-05-08T11:08:18.649Z] 1. Register with subscription-manager [2026-05-08T11:08:18.649Z] # subscription-manager register [2026-05-08T11:08:18.649Z] [2026-05-08T11:08:18.649Z] 2. Register with insights-client [2026-05-08T11:08:18.649Z] # insights-client --register [2026-05-08T11:08:19.391Z] This system is currently not registered. [2026-05-08T11:08:19.736Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-05-08T11:08:19.736Z] This might take a few seconds. [2026-05-08T11:08:19.736Z] [2026-05-08T11:08:20.081Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-08T11:08:27.514Z] 𐄂 Disconnected from Red Hat Insights [2026-05-08T11:08:27.514Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-08T11:08:27.514Z] [2026-05-08T11:08:27.514Z] Manage your connected systems: https://red.ht/connector [2026-05-08T11:08:33.562Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-08T11:08:33.562Z] [2026-05-08T11:08:33.562Z] 1. Register with subscription-manager [2026-05-08T11:08:33.562Z] # subscription-manager register [2026-05-08T11:08:33.562Z] [2026-05-08T11:08:33.562Z] 2. Register with insights-client [2026-05-08T11:08:33.562Z] # insights-client --register [2026-05-08T11:08:34.440Z] This system is currently not registered. [2026-05-08T11:08:34.784Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-05-08T11:08:34.784Z] This might take a few seconds. [2026-05-08T11:08:34.784Z] [2026-05-08T11:08:34.784Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-08T11:08:39.665Z] 𐄂 Disconnected from Red Hat Insights [2026-05-08T11:08:40.405Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-08T11:08:40.405Z] [2026-05-08T11:08:40.405Z] Manage your connected systems: https://red.ht/connector [2026-05-08T11:08:45.287Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-08T11:08:45.287Z] [2026-05-08T11:08:45.287Z] 1. Register with subscription-manager [2026-05-08T11:08:45.287Z] # subscription-manager register [2026-05-08T11:08:45.287Z] [2026-05-08T11:08:45.287Z] 2. Register with insights-client [2026-05-08T11:08:45.287Z] # insights-client --register [2026-05-08T11:08:45.632Z] This system is currently not registered. [2026-05-08T11:08:46.373Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-05-08T11:08:46.373Z] This might take a few seconds. [2026-05-08T11:08:46.373Z] [2026-05-08T11:08:46.373Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-08T11:08:51.258Z] 𐄂 Disconnected from Red Hat Insights [2026-05-08T11:08:51.997Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-08T11:08:51.997Z] [2026-05-08T11:08:51.997Z] Manage your connected systems: https://red.ht/connector [2026-05-08T11:08:56.027Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-08T11:08:56.027Z] [2026-05-08T11:08:56.027Z] 1. Register with subscription-manager [2026-05-08T11:08:56.027Z] # subscription-manager register [2026-05-08T11:08:56.027Z] [2026-05-08T11:08:56.027Z] 2. Register with insights-client [2026-05-08T11:08:56.027Z] # insights-client --register [2026-05-08T11:08:56.768Z] This system is currently not registered. [2026-05-08T11:08:57.508Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-05-08T11:08:57.508Z] This might take a few seconds. [2026-05-08T11:08:57.508Z] [2026-05-08T11:08:57.508Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-05-08T11:09:03.559Z] 𐄂 Disconnected from Red Hat Insights [2026-05-08T11:09:03.559Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-05-08T11:09:03.559Z] [2026-05-08T11:09:03.559Z] Manage your connected systems: https://red.ht/connector [2026-05-08T11:09:07.449Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-05-08T11:09:07.449Z] [2026-05-08T11:09:07.449Z] 1. Register with subscription-manager [2026-05-08T11:09:07.449Z] # subscription-manager register [2026-05-08T11:09:07.449Z] [2026-05-08T11:09:07.449Z] 2. Register with insights-client [2026-05-08T11:09:07.449Z] # insights-client --register [2026-05-08T11:09:08.644Z] This system is currently not registered. [2026-05-08T11:09:08.716Z] [Pipeline] } [2026-05-08T11:09:08.719Z] [Pipeline] // timeout [2026-05-08T11:09:08.723Z] [Pipeline] } [2026-05-08T11:09:08.725Z] [Pipeline] // script [2026-05-08T11:09:08.727Z] [Pipeline] } [2026-05-08T11:09:08.729Z] [Pipeline] // stage [2026-05-08T11:09:08.733Z] [Pipeline] stage [2026-05-08T11:09:08.733Z] [Pipeline] { (Declarative: Post Actions) [2026-05-08T11:09:08.739Z] [Pipeline] cleanWs [2026-05-08T11:09:08.788Z] [WS-CLEANUP] Deleting project workspace... [2026-05-08T11:09:08.788Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-05-08T11:09:08.813Z] [WS-CLEANUP] done [2026-05-08T11:09:08.816Z] [Pipeline] script [2026-05-08T11:09:08.818Z] [Pipeline] { [2026-05-08T11:09:08.833Z] [Pipeline] stage [2026-05-08T11:09:08.834Z] [Pipeline] { (Copying logs) [2026-05-08T11:09:08.865Z] [Pipeline] node [2026-05-08T11:09:08.956Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-05-08T11:09:08.956Z] [Pipeline] { [2026-05-08T11:09:08.990Z] [Pipeline] node [2026-05-08T11:09:09.081Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-05-08T11:09:09.082Z] [Pipeline] { [2026-05-08T11:09:09.085Z] [Pipeline] sh [2026-05-08T11:09:09.342Z] + mkdir -p /var/lib/jenkins/locks [2026-05-08T11:09:09.352Z] [Pipeline] echo [2026-05-08T11:09:09.352Z] RWLock: FD for lock log_archive in stage projectFinishUp is 404 [2026-05-08T11:09:09.355Z] [Pipeline] sleep [2026-05-08T11:09:09.355Z] No need to sleep any longer [2026-05-08T11:09:09.358Z] [Pipeline] echo [2026-05-08T11:09:09.358Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-05-08T11:09:09.361Z] [Pipeline] } [2026-05-08T11:09:09.363Z] [Pipeline] // node [2026-05-08T11:09:09.367Z] [Pipeline] echo [2026-05-08T11:09:09.367Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/486/ [2026-05-08T11:09:09.368Z] [Pipeline] sh [2026-05-08T11:09:09.623Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/486/ [2026-05-08T11:09:09.625Z] [Pipeline] sh