[2026-10-04T08:49:52.530Z] Started by upstream project "anvil/anvil-build-cron" build number 185 [2026-10-04T08:49:52.530Z] originally caused by: [2026-10-04T08:49:52.530Z] Started by timer [2026-10-04T08:49:53.663Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-10-04T08:49:53.671Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-10-04T08:49:53.883Z] Examining kronosnet/ci-tools [2026-10-04T08:49:53.883Z] Attempting to resolve main as a branch [2026-10-04T08:49:54.059Z] Resolved main as branch main at revision b175ea1eaa3b116a3499f67254edeed28a1abb28 [2026-10-04T08:49:54.060Z] Selected Git installation does not exist. Using Default [2026-10-04T08:49:54.060Z] The recommended git tool is: NONE [2026-10-04T08:49:54.060Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-10-04T08:49:54.062Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-10-04T08:49:54.062Z] Wiping out workspace first. [2026-10-04T08:49:54.084Z] Cloning the remote Git repository [2026-10-04T08:49:54.084Z] Using shallow clone with depth 1 [2026-10-04T08:49:54.084Z] Avoid fetching tags [2026-10-04T08:49:54.084Z] Honoring refspec on initial clone [2026-10-04T08:49:54.085Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-10-04T08:49:54.085Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-10-04T08:49:54.098Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-10-04T08:49:54.098Z] > git --version # timeout=10 [2026-10-04T08:49:54.107Z] > git --version # 'git version 2.52.0' [2026-10-04T08:49:54.107Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-10-04T08:49:54.114Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-10-04T08:49:54.382Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-10-04T08:49:54.389Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-10-04T08:49:54.397Z] Avoid second fetch [2026-10-04T08:49:54.398Z] Checking out Revision b175ea1eaa3b116a3499f67254edeed28a1abb28 (main) [2026-10-04T08:49:54.398Z] > git config core.sparsecheckout # timeout=10 [2026-10-04T08:49:54.405Z] > git checkout -f b175ea1eaa3b116a3499f67254edeed28a1abb28 # timeout=10 [2026-10-04T08:49:54.441Z] Commit message: "support ansible development/testing in CITBRANCH" [2026-10-04T08:49:54.451Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-10-04T08:49:54.666Z] Examining kronosnet/ci-tools [2026-10-04T08:49:54.666Z] Attempting to resolve main as a branch [2026-10-04T08:49:54.813Z] Resolved main as branch main at revision b175ea1eaa3b116a3499f67254edeed28a1abb28 [2026-10-04T08:49:54.814Z] Selected Git installation does not exist. Using Default [2026-10-04T08:49:54.814Z] The recommended git tool is: NONE [2026-10-04T08:49:54.814Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-10-04T08:49:54.817Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-10-04T08:49:54.817Z] Wiping out workspace first. [2026-10-04T08:49:54.836Z] Cloning the remote Git repository [2026-10-04T08:49:54.836Z] Using shallow clone with depth 1 [2026-10-04T08:49:54.836Z] Avoid fetching tags [2026-10-04T08:49:54.836Z] Honoring refspec on initial clone [2026-10-04T08:49:54.836Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-10-04T08:49:54.836Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-10-04T08:49:54.846Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-10-04T08:49:54.846Z] > git --version # timeout=10 [2026-10-04T08:49:54.853Z] > git --version # 'git version 2.52.0' [2026-10-04T08:49:54.853Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-10-04T08:49:54.860Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-10-04T08:49:55.119Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-10-04T08:49:55.127Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-10-04T08:49:55.137Z] Avoid second fetch [2026-10-04T08:49:55.137Z] Checking out Revision b175ea1eaa3b116a3499f67254edeed28a1abb28 (main) [2026-10-04T08:49:55.137Z] > git config core.sparsecheckout # timeout=10 [2026-10-04T08:49:55.144Z] > git checkout -f b175ea1eaa3b116a3499f67254edeed28a1abb28 # timeout=10 [2026-10-04T08:49:55.219Z] Commit message: "support ansible development/testing in CITBRANCH" [2026-10-04T08:49:55.286Z] [Pipeline] Start of Pipeline [2026-10-04T08:49:55.380Z] [Pipeline] node [2026-10-04T08:49:55.984Z] Running on anvil-ci-bm-phy04 in /srv/workspace/anvil/anvil-build-functional-testing [2026-10-04T08:49:55.984Z] [Pipeline] { [2026-10-04T08:49:55.993Z] [Pipeline] stage [2026-10-04T08:49:55.993Z] [Pipeline] { (Prep build env) [2026-10-04T08:49:56.008Z] [Pipeline] script [2026-10-04T08:49:56.010Z] [Pipeline] { [2026-10-04T08:49:56.015Z] [Pipeline] cleanWs [2026-10-04T08:49:56.065Z] [WS-CLEANUP] Deleting project workspace... [2026-10-04T08:49:56.065Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-10-04T08:49:56.089Z] [WS-CLEANUP] done [2026-10-04T08:49:56.112Z] [Pipeline] lock [2026-10-04T08:49:56.112Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-10-04T08:49:56.112Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-10-04T08:49:56.112Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-10-04T08:49:56.112Z] [Pipeline] { [2026-10-04T08:49:56.116Z] [Pipeline] } [2026-10-04T08:49:56.116Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-10-04T08:49:56.118Z] [Pipeline] // lock [2026-10-04T08:49:56.126Z] [Pipeline] } [2026-10-04T08:49:56.128Z] [Pipeline] // script [2026-10-04T08:49:56.132Z] [Pipeline] } [2026-10-04T08:49:56.136Z] [Pipeline] // stage [2026-10-04T08:49:56.139Z] [Pipeline] stage [2026-10-04T08:49:56.140Z] [Pipeline] { (Run functional testing) [2026-10-04T08:49:56.148Z] [Pipeline] script [2026-10-04T08:49:56.149Z] [Pipeline] { [2026-10-04T08:49:56.166Z] [Pipeline] echo [2026-10-04T08:49:56.166Z] runWithTimeout 360 [2026-10-04T08:49:56.167Z] [Pipeline] timeout [2026-10-04T08:49:56.167Z] Timeout set to expire in 6 hr 0 min [2026-10-04T08:49:56.167Z] [Pipeline] { [2026-10-04T08:49:56.173Z] [Pipeline] sh [2026-10-04T08:49:56.684Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 enterprise none all [2026-10-04T08:49:56.684Z] CITBRANCH: main [2026-10-04T08:49:56.684Z] CITHOME: /root/ci-tools [2026-10-04T08:49:56.684Z] CITCMD: ci-setup-anvil-bm-vm [2026-10-04T08:49:56.684Z] CITCMDARGS: rhel-9 enterprise none all [2026-10-04T08:49:56.684Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-10-04T08:49:57.028Z] Genering ssh keys and ssh config [2026-10-04T08:49:57.371Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-10-04T08:53:06.527Z] Destroying VMs (if any) [2026-10-04T08:53:06.527Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-10-04T08:53:06.527Z] Genering ssh keys and ssh config [2026-10-04T08:53:06.527Z] Cache already updated [2026-10-04T08:53:06.527Z] Creating nodes disks [2026-10-04T08:53:06.527Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-10-04T08:53:06.527Z] Generating an-test-deployX kickstart files/isos [2026-10-04T08:53:06.527Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-10-04T08:53:06.527Z] Media current: stdio file, overwriteable [2026-10-04T08:53:06.527Z] Media status : is blank [2026-10-04T08:53:06.527Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1715g free [2026-10-04T08:53:06.527Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] ISO image produced: 184 sectors [2026-10-04T08:53:06.527Z] Written to medium : 184 sectors at LBA 0 [2026-10-04T08:53:06.527Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-10-04T08:53:06.527Z] Media current: stdio file, overwriteable [2026-10-04T08:53:06.527Z] Media status : is blank [2026-10-04T08:53:06.527Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1724g free [2026-10-04T08:53:06.527Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] ISO image produced: 184 sectors [2026-10-04T08:53:06.527Z] Written to medium : 184 sectors at LBA 0 [2026-10-04T08:53:06.527Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-10-04T08:53:06.527Z] Media current: stdio file, overwriteable [2026-10-04T08:53:06.527Z] Media status : is blank [2026-10-04T08:53:06.527Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1724g free [2026-10-04T08:53:06.527Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] ISO image produced: 184 sectors [2026-10-04T08:53:06.527Z] Written to medium : 184 sectors at LBA 0 [2026-10-04T08:53:06.527Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-10-04T08:53:06.527Z] Media current: stdio file, overwriteable [2026-10-04T08:53:06.527Z] Media status : is blank [2026-10-04T08:53:06.527Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1724g free [2026-10-04T08:53:06.527Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] ISO image produced: 184 sectors [2026-10-04T08:53:06.527Z] Written to medium : 184 sectors at LBA 0 [2026-10-04T08:53:06.527Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-10-04T08:53:06.527Z] Media current: stdio file, overwriteable [2026-10-04T08:53:06.527Z] Media status : is blank [2026-10-04T08:53:06.527Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1724g free [2026-10-04T08:53:06.527Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-10-04T08:53:06.527Z] ISO image produced: 184 sectors [2026-10-04T08:53:06.527Z] Written to medium : 184 sectors at LBA 0 [2026-10-04T08:53:06.527Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-10-04T08:53:06.527Z] [2026-10-04T08:53:06.527Z] Generating cloud-init configs [2026-10-04T08:53:06.527Z] Generating cloud-init metadata and seed images [2026-10-04T08:53:06.527Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-10-04T08:53:06.527Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-10-04T08:53:06.527Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-10-04T08:53:06.527Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-10-04T08:53:06.527Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-10-04T08:53:06.527Z] Creating VMs [2026-10-04T08:53:06.872Z] [2026-10-04T08:53:06.872Z] Starting install... [2026-10-04T08:53:06.872Z] [2026-10-04T08:53:06.872Z] Creating domain... | 00:00 [2026-10-04T08:53:06.872Z] Domain creation completed. [2026-10-04T08:53:08.067Z] [2026-10-04T08:53:08.067Z] Starting install... [2026-10-04T08:53:08.067Z] [2026-10-04T08:53:08.067Z] Creating domain... | 00:00 [2026-10-04T08:53:08.411Z] Domain creation completed. [2026-10-04T08:53:09.606Z] [2026-10-04T08:53:09.606Z] Starting install... [2026-10-04T08:53:09.606Z] [2026-10-04T08:53:09.606Z] Creating domain... | 00:00 [2026-10-04T08:53:09.606Z] Domain creation completed. [2026-10-04T08:53:10.800Z] [2026-10-04T08:53:10.800Z] Starting install... [2026-10-04T08:53:10.800Z] [2026-10-04T08:53:10.800Z] Creating domain... | 00:00 [2026-10-04T08:53:10.800Z] Domain creation completed. [2026-10-04T08:53:12.523Z] [2026-10-04T08:53:12.523Z] Starting install... [2026-10-04T08:53:12.523Z] [2026-10-04T08:53:12.523Z] Creating domain... | 00:00 [2026-10-04T08:53:12.523Z] Domain creation completed. [2026-10-04T08:53:12.523Z] Waiting for nodes and cloud-init to complete full init [2026-10-04T08:53:15.593Z] cloud-init status node an-striker01: [2026-10-04T08:53:18.644Z] cloud-init status node an-striker02: [2026-10-04T08:53:21.693Z] cloud-init status node an-a01n01: [2026-10-04T08:53:24.743Z] cloud-init status node an-a01n02: [2026-10-04T08:53:28.632Z] cloud-init status node an-a01dr01: [2026-10-04T08:53:28.632Z] Sun Oct 4 04:53:27 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:53:39.657Z] cloud-init status node an-striker01: running [2026-10-04T08:53:42.346Z] cloud-init status node an-striker02: running [2026-10-04T08:53:42.346Z] cloud-init status node an-a01n01: running [2026-10-04T08:53:42.939Z] cloud-init status node an-a01n02: running [2026-10-04T08:53:44.109Z] cloud-init status node an-a01dr01: running [2026-10-04T08:53:44.109Z] Sun Oct 4 04:53:43 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:53:54.540Z] cloud-init status node an-striker01: running [2026-10-04T08:53:55.597Z] cloud-init status node an-striker02: running [2026-10-04T08:53:56.053Z] cloud-init status node an-a01n01: running [2026-10-04T08:53:57.247Z] cloud-init status node an-a01n02: running [2026-10-04T08:54:02.621Z] cloud-init status node an-a01dr01: running [2026-10-04T08:54:02.621Z] Sun Oct 4 04:54:02 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:54:19.202Z] cloud-init status node an-striker01: running [2026-10-04T08:54:19.202Z] cloud-init status node an-striker02: running [2026-10-04T08:54:21.001Z] cloud-init status node an-a01n01: running [2026-10-04T08:54:21.001Z] cloud-init status node an-a01n02: running [2026-10-04T08:54:22.461Z] cloud-init status node an-a01dr01: running [2026-10-04T08:54:22.461Z] Sun Oct 4 04:54:21 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:54:33.626Z] cloud-init status node an-striker01: running [2026-10-04T08:54:33.626Z] cloud-init status node an-striker02: running [2026-10-04T08:54:34.416Z] cloud-init status node an-a01n01: running [2026-10-04T08:54:40.312Z] cloud-init status node an-a01n02: running [2026-10-04T08:54:41.127Z] cloud-init status node an-a01dr01: running [2026-10-04T08:54:41.127Z] Sun Oct 4 04:54:40 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:54:51.833Z] cloud-init status node an-striker01: running [2026-10-04T08:54:54.692Z] cloud-init status node an-striker02: running [2026-10-04T08:54:54.692Z] cloud-init status node an-a01n01: running [2026-10-04T08:54:54.692Z] cloud-init status node an-a01n02: running [2026-10-04T08:54:55.742Z] cloud-init status node an-a01dr01: running [2026-10-04T08:54:55.742Z] Sun Oct 4 04:54:55 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:55:06.709Z] cloud-init status node an-striker01: running [2026-10-04T08:55:11.978Z] cloud-init status node an-striker02: running [2026-10-04T08:55:12.980Z] cloud-init status node an-a01n01: running [2026-10-04T08:55:14.198Z] cloud-init status node an-a01n02: running [2026-10-04T08:55:14.738Z] cloud-init status node an-a01dr01: running [2026-10-04T08:55:14.738Z] Sun Oct 4 04:55:14 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:55:25.851Z] cloud-init status node an-striker01: running [2026-10-04T08:55:26.462Z] cloud-init status node an-striker02: running [2026-10-04T08:55:27.300Z] cloud-init status node an-a01n01: running [2026-10-04T08:55:28.893Z] cloud-init status node an-a01n02: running [2026-10-04T08:55:28.893Z] cloud-init status node an-a01dr01: running [2026-10-04T08:55:28.893Z] Sun Oct 4 04:55:28 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:55:57.772Z] cloud-init status node an-striker01: running [2026-10-04T08:55:57.772Z] cloud-init status node an-striker02: running [2026-10-04T08:55:57.772Z] cloud-init status node an-a01n01: running [2026-10-04T08:55:57.772Z] cloud-init status node an-a01n02: running [2026-10-04T08:56:08.306Z] cloud-init status node an-a01dr01: [2026-10-04T08:56:08.306Z] Sun Oct 4 04:56:07 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:56:18.484Z] cloud-init status node an-striker01: running [2026-10-04T08:56:19.404Z] cloud-init status node an-striker02: running [2026-10-04T08:56:20.325Z] cloud-init status node an-a01n01: running [2026-10-04T08:56:21.111Z] cloud-init status node an-a01n02: running [2026-10-04T08:56:21.925Z] cloud-init status node an-a01dr01: running [2026-10-04T08:56:21.925Z] Sun Oct 4 04:56:21 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:56:34.588Z] cloud-init status node an-striker01: running [2026-10-04T08:56:34.588Z] cloud-init status node an-striker02: running [2026-10-04T08:56:34.588Z] cloud-init status node an-a01n01: running [2026-10-04T08:56:35.521Z] cloud-init status node an-a01n02: running [2026-10-04T08:56:37.087Z] cloud-init status node an-a01dr01: running [2026-10-04T08:56:37.087Z] Sun Oct 4 04:56:36 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:56:47.185Z] cloud-init status node an-striker01: running [2026-10-04T08:56:48.191Z] cloud-init status node an-striker02: running [2026-10-04T08:56:49.755Z] cloud-init status node an-a01n01: running [2026-10-04T08:56:49.755Z] cloud-init status node an-a01n02: running [2026-10-04T08:56:50.849Z] cloud-init status node an-a01dr01: running [2026-10-04T08:56:50.849Z] Sun Oct 4 04:56:50 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:57:02.705Z] cloud-init status node an-striker01: running [2026-10-04T08:57:02.705Z] cloud-init status node an-striker02: running [2026-10-04T08:57:03.166Z] cloud-init status node an-a01n01: running [2026-10-04T08:57:04.413Z] cloud-init status node an-a01n02: running [2026-10-04T08:57:05.053Z] cloud-init status node an-a01dr01: running [2026-10-04T08:57:05.053Z] Sun Oct 4 04:57:04 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:57:17.783Z] cloud-init status node an-striker01: running [2026-10-04T08:57:27.368Z] cloud-init status node an-striker02: [2026-10-04T08:57:27.368Z] cloud-init status node an-a01n01: running [2026-10-04T08:57:27.736Z] cloud-init status node an-a01n02: running [2026-10-04T08:57:37.688Z] cloud-init status node an-a01dr01: [2026-10-04T08:57:37.688Z] Sun Oct 4 04:57:37 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:57:48.552Z] cloud-init status node an-striker01: running [2026-10-04T08:57:49.751Z] cloud-init status node an-striker02: running [2026-10-04T08:57:50.735Z] cloud-init status node an-a01n01: running [2026-10-04T08:57:51.143Z] cloud-init status node an-a01n02: running [2026-10-04T08:57:52.683Z] cloud-init status node an-a01dr01: running [2026-10-04T08:57:52.683Z] Sun Oct 4 04:57:51 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:58:04.237Z] cloud-init status node an-striker01: running [2026-10-04T08:58:04.237Z] cloud-init status node an-striker02: running [2026-10-04T08:58:14.950Z] cloud-init status node an-a01n01: running [2026-10-04T08:58:16.277Z] cloud-init status node an-a01n02: running [2026-10-04T08:58:16.277Z] cloud-init status node an-a01dr01: running [2026-10-04T08:58:16.277Z] Sun Oct 4 04:58:16 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:58:28.294Z] cloud-init status node an-striker01: running [2026-10-04T08:58:28.294Z] cloud-init status node an-striker02: running [2026-10-04T08:58:31.126Z] cloud-init status node an-a01n01: running [2026-10-04T08:58:31.126Z] cloud-init status node an-a01n02: running [2026-10-04T08:58:36.144Z] cloud-init status node an-a01dr01: running [2026-10-04T08:58:36.144Z] Sun Oct 4 04:58:35 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:58:46.360Z] cloud-init status node an-striker01: running [2026-10-04T08:58:47.144Z] cloud-init status node an-striker02: running [2026-10-04T08:58:48.729Z] cloud-init status node an-a01n01: running [2026-10-04T08:58:50.666Z] cloud-init status node an-a01n02: running [2026-10-04T08:58:50.666Z] cloud-init status node an-a01dr01: running [2026-10-04T08:58:50.666Z] Sun Oct 4 04:58:49 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:59:01.655Z] cloud-init status node an-striker01: running [2026-10-04T08:59:01.655Z] cloud-init status node an-striker02: running [2026-10-04T08:59:02.574Z] cloud-init status node an-a01n01: running [2026-10-04T08:59:03.481Z] cloud-init status node an-a01n02: running [2026-10-04T08:59:04.422Z] cloud-init status node an-a01dr01: running [2026-10-04T08:59:04.422Z] Sun Oct 4 04:59:04 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:59:17.038Z] cloud-init status node an-striker01: running [2026-10-04T08:59:21.386Z] cloud-init status node an-striker02: running [2026-10-04T08:59:21.782Z] cloud-init status node an-a01n01: running [2026-10-04T08:59:23.068Z] cloud-init status node an-a01n02: running [2026-10-04T08:59:24.007Z] cloud-init status node an-a01dr01: running [2026-10-04T08:59:24.007Z] Sun Oct 4 04:59:23 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:59:35.111Z] cloud-init status node an-striker01: running [2026-10-04T08:59:35.600Z] cloud-init status node an-striker02: running [2026-10-04T08:59:36.825Z] cloud-init status node an-a01n01: running [2026-10-04T08:59:37.226Z] cloud-init status node an-a01n02: running [2026-10-04T08:59:38.085Z] cloud-init status node an-a01dr01: running [2026-10-04T08:59:38.085Z] Sun Oct 4 04:59:38 EDT 2026 actual: 0 expected: 5 [2026-10-04T08:59:50.802Z] cloud-init status node an-striker01: running [2026-10-04T08:59:50.802Z] cloud-init status node an-striker02: running [2026-10-04T08:59:50.802Z] cloud-init status node an-a01n01: running [2026-10-04T08:59:52.273Z] cloud-init status node an-a01n02: running [2026-10-04T08:59:53.552Z] cloud-init status node an-a01dr01: running [2026-10-04T08:59:53.552Z] Sun Oct 4 04:59:52 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:00:04.958Z] cloud-init status node an-striker01: running [2026-10-04T09:00:04.958Z] cloud-init status node an-striker02: running [2026-10-04T09:00:05.429Z] cloud-init status node an-a01n01: running [2026-10-04T09:00:06.842Z] cloud-init status node an-a01n02: running [2026-10-04T09:00:07.393Z] cloud-init status node an-a01dr01: running [2026-10-04T09:00:07.393Z] Sun Oct 4 05:00:07 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:00:18.600Z] cloud-init status node an-striker01: running [2026-10-04T09:00:19.337Z] cloud-init status node an-striker02: running [2026-10-04T09:00:20.076Z] cloud-init status node an-a01n01: running [2026-10-04T09:00:21.269Z] cloud-init status node an-a01n02: running [2026-10-04T09:00:22.008Z] cloud-init status node an-a01dr01: running [2026-10-04T09:00:22.008Z] Sun Oct 4 05:00:21 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:00:33.009Z] cloud-init status node an-striker01: running [2026-10-04T09:00:33.747Z] cloud-init status node an-striker02: running [2026-10-04T09:00:34.486Z] cloud-init status node an-a01n01: running [2026-10-04T09:00:35.679Z] cloud-init status node an-a01n02: running [2026-10-04T09:00:36.417Z] cloud-init status node an-a01dr01: running [2026-10-04T09:00:36.417Z] Sun Oct 4 05:00:36 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:00:47.438Z] cloud-init status node an-striker01: running [2026-10-04T09:00:47.783Z] cloud-init status node an-striker02: running [2026-10-04T09:00:48.977Z] cloud-init status node an-a01n01: running [2026-10-04T09:00:49.716Z] cloud-init status node an-a01n02: running [2026-10-04T09:00:50.456Z] cloud-init status node an-a01dr01: running [2026-10-04T09:00:50.456Z] Sun Oct 4 05:00:50 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:01:01.459Z] cloud-init status node an-striker01: running [2026-10-04T09:01:02.197Z] cloud-init status node an-striker02: running [2026-10-04T09:01:03.390Z] cloud-init status node an-a01n01: running [2026-10-04T09:01:04.128Z] cloud-init status node an-a01n02: running [2026-10-04T09:01:04.867Z] cloud-init status node an-a01dr01: running [2026-10-04T09:01:04.867Z] Sun Oct 4 05:01:04 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:01:15.867Z] cloud-init status node an-striker01: running [2026-10-04T09:01:16.606Z] cloud-init status node an-striker02: running [2026-10-04T09:01:17.800Z] cloud-init status node an-a01n01: running [2026-10-04T09:01:18.538Z] cloud-init status node an-a01n02: running [2026-10-04T09:01:19.276Z] cloud-init status node an-a01dr01: running [2026-10-04T09:01:19.276Z] Sun Oct 4 05:01:19 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:01:32.610Z] cloud-init status node an-striker01: running [2026-10-04T09:01:32.610Z] cloud-init status node an-striker02: running [2026-10-04T09:01:32.610Z] cloud-init status node an-a01n01: running [2026-10-04T09:01:32.954Z] cloud-init status node an-a01n02: running [2026-10-04T09:01:34.147Z] cloud-init status node an-a01dr01: running [2026-10-04T09:01:34.147Z] Sun Oct 4 05:01:33 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:01:45.149Z] cloud-init status node an-striker01: running [2026-10-04T09:01:45.887Z] cloud-init status node an-striker02: running [2026-10-04T09:01:46.625Z] cloud-init status node an-a01n01: running [2026-10-04T09:01:47.363Z] cloud-init status node an-a01n02: running [2026-10-04T09:01:48.556Z] cloud-init status node an-a01dr01: running [2026-10-04T09:01:48.556Z] Sun Oct 4 05:01:48 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:01:59.559Z] cloud-init status node an-striker01: running [2026-10-04T09:02:00.299Z] cloud-init status node an-striker02: running [2026-10-04T09:02:01.039Z] cloud-init status node an-a01n01: running [2026-10-04T09:02:01.776Z] cloud-init status node an-a01n02: running [2026-10-04T09:02:02.971Z] cloud-init status node an-a01dr01: running [2026-10-04T09:02:02.971Z] Sun Oct 4 05:02:02 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:02:13.980Z] cloud-init status node an-striker01: running [2026-10-04T09:02:14.721Z] cloud-init status node an-striker02: running [2026-10-04T09:02:15.461Z] cloud-init status node an-a01n01: running [2026-10-04T09:02:16.200Z] cloud-init status node an-a01n02: running [2026-10-04T09:02:17.402Z] cloud-init status node an-a01dr01: running [2026-10-04T09:02:17.402Z] Sun Oct 4 05:02:17 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:02:28.407Z] cloud-init status node an-striker01: running [2026-10-04T09:02:29.145Z] cloud-init status node an-striker02: running [2026-10-04T09:02:29.883Z] cloud-init status node an-a01n01: running [2026-10-04T09:02:31.077Z] cloud-init status node an-a01n02: running [2026-10-04T09:02:31.816Z] cloud-init status node an-a01dr01: running [2026-10-04T09:02:31.816Z] Sun Oct 4 05:02:31 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:02:42.817Z] cloud-init status node an-striker01: running [2026-10-04T09:02:43.557Z] cloud-init status node an-striker02: running [2026-10-04T09:02:44.296Z] cloud-init status node an-a01n01: running [2026-10-04T09:02:45.490Z] cloud-init status node an-a01n02: running [2026-10-04T09:02:46.235Z] cloud-init status node an-a01dr01: running [2026-10-04T09:02:46.235Z] Sun Oct 4 05:02:46 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:02:57.236Z] cloud-init status node an-striker01: running [2026-10-04T09:02:57.975Z] cloud-init status node an-striker02: running [2026-10-04T09:02:58.714Z] cloud-init status node an-a01n01: running [2026-10-04T09:02:59.920Z] cloud-init status node an-a01n02: running [2026-10-04T09:03:00.658Z] cloud-init status node an-a01dr01: running [2026-10-04T09:03:00.658Z] Sun Oct 4 05:03:00 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:03:11.657Z] cloud-init status node an-striker01: running [2026-10-04T09:03:12.397Z] cloud-init status node an-striker02: running [2026-10-04T09:03:14.117Z] cloud-init status node an-a01n01: running [2026-10-04T09:03:15.311Z] cloud-init status node an-a01n02: running [2026-10-04T09:03:17.033Z] cloud-init status node an-a01dr01: running [2026-10-04T09:03:17.033Z] Sun Oct 4 05:03:16 EDT 2026 actual: 0 expected: 5 [2026-10-04T09:03:28.046Z] cloud-init status node an-striker01: running [2026-10-04T09:03:29.261Z] cloud-init status node an-striker02: running [2026-10-04T09:03:30.001Z] cloud-init status node an-a01n01: done [2026-10-04T09:03:31.196Z] cloud-init status node an-a01n02: done [2026-10-04T09:03:32.394Z] cloud-init status node an-a01dr01: done [2026-10-04T09:03:32.394Z] Sun Oct 4 05:03:31 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:03:43.657Z] cloud-init status node an-striker01: running [2026-10-04T09:03:44.492Z] cloud-init status node an-striker02: running [2026-10-04T09:03:50.564Z] cloud-init status node an-a01n01: done [2026-10-04T09:03:51.760Z] cloud-init status node an-a01n02: done [2026-10-04T09:03:52.500Z] cloud-init status node an-a01dr01: done [2026-10-04T09:03:52.500Z] Sun Oct 4 05:03:52 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:04:03.508Z] cloud-init status node an-striker01: running [2026-10-04T09:04:04.701Z] cloud-init status node an-striker02: running [2026-10-04T09:04:05.441Z] cloud-init status node an-a01n01: done [2026-10-04T09:04:06.635Z] cloud-init status node an-a01n02: done [2026-10-04T09:04:07.373Z] cloud-init status node an-a01dr01: done [2026-10-04T09:04:07.373Z] Sun Oct 4 05:04:07 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:04:18.388Z] cloud-init status node an-striker01: running [2026-10-04T09:04:19.127Z] cloud-init status node an-striker02: running [2026-10-04T09:04:20.411Z] cloud-init status node an-a01n01: done [2026-10-04T09:04:21.149Z] cloud-init status node an-a01n02: done [2026-10-04T09:04:21.885Z] cloud-init status node an-a01dr01: done [2026-10-04T09:04:21.885Z] Sun Oct 4 05:04:21 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:04:35.199Z] cloud-init status node an-striker01: running [2026-10-04T09:04:35.199Z] cloud-init status node an-striker02: running [2026-10-04T09:04:35.199Z] cloud-init status node an-a01n01: done [2026-10-04T09:04:35.937Z] cloud-init status node an-a01n02: done [2026-10-04T09:04:37.129Z] cloud-init status node an-a01dr01: done [2026-10-04T09:04:37.129Z] Sun Oct 4 05:04:36 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:04:48.129Z] cloud-init status node an-striker01: running [2026-10-04T09:04:49.321Z] cloud-init status node an-striker02: running [2026-10-04T09:04:50.060Z] cloud-init status node an-a01n01: done [2026-10-04T09:04:51.253Z] cloud-init status node an-a01n02: done [2026-10-04T09:04:51.992Z] cloud-init status node an-a01dr01: done [2026-10-04T09:04:51.992Z] Sun Oct 4 05:04:51 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:05:03.018Z] cloud-init status node an-striker01: running [2026-10-04T09:05:04.209Z] cloud-init status node an-striker02: running [2026-10-04T09:05:05.403Z] cloud-init status node an-a01n01: done [2026-10-04T09:05:06.141Z] cloud-init status node an-a01n02: done [2026-10-04T09:05:07.333Z] cloud-init status node an-a01dr01: done [2026-10-04T09:05:07.333Z] Sun Oct 4 05:05:06 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:05:18.332Z] cloud-init status node an-striker01: running [2026-10-04T09:05:19.525Z] cloud-init status node an-striker02: running [2026-10-04T09:05:20.263Z] cloud-init status node an-a01n01: done [2026-10-04T09:05:21.001Z] cloud-init status node an-a01n02: done [2026-10-04T09:05:22.195Z] cloud-init status node an-a01dr01: done [2026-10-04T09:05:22.195Z] Sun Oct 4 05:05:21 EDT 2026 actual: 3 expected: 5 [2026-10-04T09:05:33.201Z] cloud-init status node an-striker01: running [2026-10-04T09:05:33.939Z] cloud-init status node an-striker02: done [2026-10-04T09:05:34.677Z] cloud-init status node an-a01n01: done [2026-10-04T09:05:35.870Z] cloud-init status node an-a01n02: done [2026-10-04T09:05:36.609Z] cloud-init status node an-a01dr01: done [2026-10-04T09:05:36.609Z] Sun Oct 4 05:05:36 EDT 2026 actual: 4 expected: 5 [2026-10-04T09:05:49.946Z] cloud-init status node an-striker01: running [2026-10-04T09:05:49.946Z] cloud-init status node an-striker02: done [2026-10-04T09:05:49.946Z] cloud-init status node an-a01n01: done [2026-10-04T09:05:51.142Z] cloud-init status node an-a01n02: done [2026-10-04T09:05:51.882Z] cloud-init status node an-a01dr01: done [2026-10-04T09:05:51.882Z] Sun Oct 4 05:05:51 EDT 2026 actual: 4 expected: 5 [2026-10-04T09:06:03.143Z] cloud-init status node an-striker01: done [2026-10-04T09:06:04.337Z] cloud-init status node an-striker02: done [2026-10-04T09:06:05.076Z] cloud-init status node an-a01n01: done [2026-10-04T09:06:06.062Z] cloud-init status node an-a01n02: done [2026-10-04T09:06:06.801Z] cloud-init status node an-a01dr01: done [2026-10-04T09:06:06.801Z] Sun Oct 4 05:06:06 EDT 2026 actual: 5 expected: 5 [2026-10-04T09:06:06.801Z] Building simengine model [2026-10-04T09:06:22.898Z] Generating anvil striker auto config [2026-10-04T09:06:23.241Z] Cloning into 'anvil-test-suite'... [2026-10-04T09:06:23.980Z] generated ansible config: [2026-10-04T09:06:23.980Z] ========================= [2026-10-04T09:06:23.980Z] [defaults] [2026-10-04T09:06:23.980Z] inventory = ansible-inventory [2026-10-04T09:06:23.980Z] forks = 500 [2026-10-04T09:06:23.980Z] any_errors_fatal = true [2026-10-04T09:06:23.980Z] display_args_to_stdout = true [2026-10-04T09:06:23.980Z] [2026-10-04T09:06:23.980Z] [ssh_connection] [2026-10-04T09:06:23.980Z] retries = 20 [2026-10-04T09:06:23.981Z] ========================= [2026-10-04T09:06:23.981Z] [2026-10-04T09:06:23.981Z] generated ansible inventory: [2026-10-04T09:06:23.981Z] ============================ [2026-10-04T09:06:23.981Z] [strikers] [2026-10-04T09:06:23.981Z] an-striker01 [2026-10-04T09:06:23.981Z] an-striker02 [2026-10-04T09:06:23.981Z] [2026-10-04T09:06:23.981Z] [nodes] [2026-10-04T09:06:23.981Z] an-a01n01 [2026-10-04T09:06:23.981Z] an-a01n02 [2026-10-04T09:06:23.981Z] [2026-10-04T09:06:23.981Z] [drnodes] [2026-10-04T09:06:23.981Z] ============================ [2026-10-04T09:06:23.981Z] [2026-10-04T09:06:23.981Z] Checking connectivity to nodes [2026-10-04T09:06:23.981Z] [2026-10-04T09:06:26.311Z] an-a01n01 | SUCCESS => { [2026-10-04T09:06:26.311Z] "ansible_facts": { [2026-10-04T09:06:26.311Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-10-04T09:06:26.311Z] }, [2026-10-04T09:06:26.311Z] "changed": false, [2026-10-04T09:06:26.311Z] "ping": "pong" [2026-10-04T09:06:26.311Z] } [2026-10-04T09:06:26.311Z] an-a01n02 | SUCCESS => { [2026-10-04T09:06:26.311Z] "ansible_facts": { [2026-10-04T09:06:26.311Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-10-04T09:06:26.311Z] }, [2026-10-04T09:06:26.311Z] "changed": false, [2026-10-04T09:06:26.311Z] "ping": "pong" [2026-10-04T09:06:26.311Z] } [2026-10-04T09:06:26.311Z] an-striker01 | SUCCESS => { [2026-10-04T09:06:26.311Z] "ansible_facts": { [2026-10-04T09:06:26.311Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-10-04T09:06:26.311Z] }, [2026-10-04T09:06:26.311Z] "changed": false, [2026-10-04T09:06:26.311Z] "ping": "pong" [2026-10-04T09:06:26.311Z] } [2026-10-04T09:06:26.311Z] an-striker02 | SUCCESS => { [2026-10-04T09:06:26.311Z] "ansible_facts": { [2026-10-04T09:06:26.311Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-10-04T09:06:26.311Z] }, [2026-10-04T09:06:26.311Z] "changed": false, [2026-10-04T09:06:26.311Z] "ping": "pong" [2026-10-04T09:06:26.311Z] } [2026-10-04T09:06:26.311Z] [2026-10-04T09:06:27.050Z] [2026-10-04T09:06:27.050Z] PLAY [Setup all nodes] ********************************************************* [2026-10-04T09:06:27.050Z] [2026-10-04T09:06:27.050Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:06:29.096Z] ok: [an-a01n01] [2026-10-04T09:06:29.096Z] ok: [an-a01n02] [2026-10-04T09:06:29.096Z] ok: [an-striker01] [2026-10-04T09:06:29.096Z] ok: [an-striker02] [2026-10-04T09:06:29.096Z] [2026-10-04T09:06:29.096Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-10-04T09:06:30.311Z] changed: [an-a01n01] [2026-10-04T09:06:30.311Z] changed: [an-a01n02] [2026-10-04T09:06:30.311Z] changed: [an-striker01] [2026-10-04T09:06:30.311Z] changed: [an-striker02] [2026-10-04T09:06:30.311Z] [2026-10-04T09:06:30.311Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-10-04T09:06:31.049Z] ok: [an-a01n01] [2026-10-04T09:06:31.049Z] ok: [an-a01n02] [2026-10-04T09:06:31.049Z] ok: [an-striker01] [2026-10-04T09:06:31.049Z] ok: [an-striker02] [2026-10-04T09:06:31.049Z] [2026-10-04T09:06:31.049Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-10-04T09:06:32.243Z] changed: [an-a01n01] [2026-10-04T09:06:32.243Z] changed: [an-a01n02] [2026-10-04T09:06:32.243Z] changed: [an-striker01] [2026-10-04T09:06:32.243Z] changed: [an-striker02] [2026-10-04T09:06:32.243Z] [2026-10-04T09:06:32.243Z] PLAY [setup strikers] ********************************************************** [2026-10-04T09:06:32.243Z] [2026-10-04T09:06:32.243Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:06:33.964Z] ok: [an-striker01] [2026-10-04T09:06:33.964Z] ok: [an-striker02] [2026-10-04T09:06:33.964Z] [2026-10-04T09:06:33.964Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-10-04T09:06:35.682Z] changed: [an-striker01] [2026-10-04T09:06:35.682Z] changed: [an-striker02] [2026-10-04T09:06:35.682Z] [2026-10-04T09:06:35.682Z] PLAY [setup lvm storage] ******************************************************* [2026-10-04T09:06:35.682Z] [2026-10-04T09:06:35.682Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:06:37.400Z] ok: [an-a01n02] [2026-10-04T09:06:37.400Z] ok: [an-a01n01] [2026-10-04T09:06:37.400Z] [2026-10-04T09:06:37.400Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-10-04T09:06:37.400Z] ] **************** [2026-10-04T09:06:38.139Z] ok: [an-a01n01] [2026-10-04T09:06:38.139Z] ok: [an-a01n02] [2026-10-04T09:06:38.139Z] [2026-10-04T09:06:38.139Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-10-04T09:06:38.139Z] skipping: [an-a01n01] [2026-10-04T09:06:38.139Z] [2026-10-04T09:06:38.139Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-10-04T09:06:38.878Z] changed: [an-a01n02] [2026-10-04T09:06:38.878Z] changed: [an-a01n01] [2026-10-04T09:06:38.878Z] [2026-10-04T09:06:38.878Z] PLAY [deploy anvil] ************************************************************ [2026-10-04T09:06:38.878Z] [2026-10-04T09:06:38.878Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:06:41.212Z] ok: [an-striker01] [2026-10-04T09:06:41.212Z] ok: [an-striker02] [2026-10-04T09:06:41.212Z] [2026-10-04T09:06:41.212Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-10-04T09:06:41.951Z] ok: [an-striker02] [2026-10-04T09:06:41.951Z] ok: [an-striker01] [2026-10-04T09:06:41.951Z] [2026-10-04T09:06:41.951Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-10-04T09:06:43.670Z] changed: [an-striker02] [2026-10-04T09:06:43.670Z] changed: [an-striker01] [2026-10-04T09:06:43.670Z] [2026-10-04T09:06:43.670Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-10-04T09:06:43.670Z] ] *** [2026-10-04T09:06:46.002Z] changed: [an-striker01] [2026-10-04T09:06:46.002Z] changed: [an-striker02] [2026-10-04T09:06:46.002Z] [2026-10-04T09:06:46.002Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:06:46.002Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:06:46.002Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-10-04T09:06:46.002Z] ] *** [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-10-04T09:13:11.330Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-10-04T09:13:11.331Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-10-04T09:13:44.962Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-10-04T09:13:44.962Z] ok: [an-striker01 -> localhost] [2026-10-04T09:13:44.962Z] ok: [an-striker02 -> localhost] [2026-10-04T09:13:44.962Z] [2026-10-04T09:13:44.962Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-10-04T09:13:44.962Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-10-04T09:13:44.962Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-10-04T09:13:44.962Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-10-04T09:13:44.962Z] echo "DB has completed sync" [2026-10-04T09:13:44.962Z] exit 0 [2026-10-04T09:13:44.962Z] fi [2026-10-04T09:13:44.962Z] if [ "$dbcon" != 2 ]; then [2026-10-04T09:13:44.962Z] echo "Detected $dbcon db connections (expected 2)" [2026-10-04T09:13:44.962Z] exit 1 [2026-10-04T09:13:44.962Z] fi [2026-10-04T09:13:44.962Z] ] *** [2026-10-04T09:14:18.593Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-10-04T09:14:18.593Z] ok: [an-striker01] [2026-10-04T09:14:18.593Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-10-04T09:14:18.593Z] ok: [an-striker02] [2026-10-04T09:14:18.593Z] [2026-10-04T09:14:18.593Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:14:18.593Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:14:18.593Z] --username=postgres anvil -t \ [2026-10-04T09:14:18.593Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-10-04T09:14:18.593Z] grep altee | \ [2026-10-04T09:14:18.593Z] grep 1$ | \ [2026-10-04T09:14:18.593Z] wc -l [2026-10-04T09:14:18.593Z] ] *** [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-10-04T09:19:58.636Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (64 retries left). [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (64 retries left). [2026-10-04T09:24:23.202Z] ok: [an-striker02 -> localhost] [2026-10-04T09:24:23.202Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (63 retries left). [2026-10-04T09:24:23.202Z] ok: [an-striker01 -> localhost] [2026-10-04T09:24:23.202Z] [2026-10-04T09:24:23.202Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:24:23.202Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:24:23.202Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-10-04T09:24:23.202Z] grep altee | \ [2026-10-04T09:24:23.202Z] wc -l [2026-10-04T09:24:23.202Z] ] *** [2026-10-04T09:24:23.202Z] ok: [an-striker02 -> localhost] [2026-10-04T09:24:23.202Z] ok: [an-striker01 -> localhost] [2026-10-04T09:24:23.202Z] [2026-10-04T09:24:23.202Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:24:23.202Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:24:23.202Z] --username=postgres anvil -t \ [2026-10-04T09:24:23.202Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-10-04T09:24:23.202Z] grep altee | \ [2026-10-04T09:24:23.202Z] grep 1$ | \ [2026-10-04T09:24:23.202Z] wc -l [2026-10-04T09:24:23.202Z] ] *** [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-10-04T09:26:31.938Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (348 retries left). [2026-10-04T09:26:31.938Z] ok: [an-striker01 -> localhost] [2026-10-04T09:26:31.938Z] ok: [an-striker02 -> localhost] [2026-10-04T09:26:31.938Z] [2026-10-04T09:26:31.938Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:26:31.938Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:26:31.938Z] --username=postgres anvil -t \ [2026-10-04T09:26:31.938Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-10-04T09:26:31.938Z] grep altee | \ [2026-10-04T09:26:31.938Z] grep 1$ | \ [2026-10-04T09:26:31.938Z] wc -l [2026-10-04T09:26:31.938Z] ] *** [2026-10-04T09:26:49.307Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-10-04T09:26:49.307Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-10-04T09:26:49.307Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-10-04T09:26:49.307Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-10-04T09:26:49.307Z] ok: [an-striker02 -> localhost] [2026-10-04T09:26:49.307Z] ok: [an-striker01 -> localhost] [2026-10-04T09:26:49.307Z] [2026-10-04T09:26:49.307Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-10-04T09:26:49.307Z] psql --host="{{ inventory_hostname }}" \ [2026-10-04T09:26:49.307Z] --username=postgres anvil -t \ [2026-10-04T09:26:49.307Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-10-04T09:26:49.307Z] grep altee | \ [2026-10-04T09:26:49.307Z] grep 1$ | \ [2026-10-04T09:26:49.307Z] wc -l [2026-10-04T09:26:49.307Z] ] *** [2026-10-04T09:26:49.307Z] ok: [an-striker01 -> localhost] [2026-10-04T09:26:49.307Z] ok: [an-striker02 -> localhost] [2026-10-04T09:26:49.307Z] [2026-10-04T09:26:49.307Z] PLAY [check network status after deployment] *********************************** [2026-10-04T09:26:49.307Z] [2026-10-04T09:26:49.307Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:26:50.752Z] ok: [an-a01n01] [2026-10-04T09:26:50.752Z] ok: [an-striker02] [2026-10-04T09:26:50.752Z] ok: [an-a01n02] [2026-10-04T09:26:50.752Z] ok: [an-striker01] [2026-10-04T09:26:50.752Z] [2026-10-04T09:26:50.752Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-10-04T09:26:50.752Z] if [ "$device" = "" ]; then [2026-10-04T09:26:50.752Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-10-04T09:26:50.752Z] exit 1 [2026-10-04T09:26:50.752Z] fi [2026-10-04T09:26:50.752Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-10-04T09:26:50.752Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-10-04T09:26:50.752Z] exit 1 [2026-10-04T09:26:50.752Z] fi [2026-10-04T09:26:50.752Z] exit 0 [2026-10-04T09:26:50.752Z] ] *** [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=ifn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=bcn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=bcn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=sn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=sn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=ifn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-10-04T09:26:58.772Z] skipping: [an-striker02] => (item=lo) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=sn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-10-04T09:26:58.772Z] skipping: [an-a01n01] => (item=lo) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=bcn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=ifn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=ifn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-10-04T09:26:58.772Z] skipping: [an-a01n02] => (item=lo) [2026-10-04T09:26:58.772Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=bcn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=sn1_link1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-10-04T09:26:58.772Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-10-04T09:26:58.772Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-10-04T09:26:58.772Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-10-04T09:26:58.772Z] skipping: [an-striker01] => (item=lo) [2026-10-04T09:26:58.772Z] [2026-10-04T09:26:58.772Z] PLAY [check pacemaker status after deployment] ********************************* [2026-10-04T09:26:58.772Z] [2026-10-04T09:26:58.772Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:26:59.510Z] ok: [an-a01n02] [2026-10-04T09:26:59.510Z] ok: [an-a01n01] [2026-10-04T09:26:59.510Z] [2026-10-04T09:26:59.510Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-10-04T09:26:59.854Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T09:26:59.854Z] [2026-10-04T09:26:59.854Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T09:26:59.854Z] ] *** [2026-10-04T09:27:03.744Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T09:27:03.744Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T09:27:03.744Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T09:27:03.744Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T09:27:03.744Z] [2026-10-04T09:27:03.744Z] PLAY [check drbd kmod] ********************************************************* [2026-10-04T09:27:03.744Z] [2026-10-04T09:27:03.744Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:27:05.465Z] ok: [an-a01n01] [2026-10-04T09:27:05.465Z] ok: [an-a01n02] [2026-10-04T09:27:05.465Z] [2026-10-04T09:27:05.465Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-10-04T09:27:05.465Z] ] *** [2026-10-04T09:27:05.810Z] ok: [an-a01n01] [2026-10-04T09:27:05.810Z] ok: [an-a01n02] [2026-10-04T09:27:05.810Z] [2026-10-04T09:27:05.810Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-10-04T09:27:05.810Z] ] *********************** [2026-10-04T09:27:05.810Z] skipping: [an-a01n01] [2026-10-04T09:27:05.810Z] skipping: [an-a01n02] [2026-10-04T09:27:05.810Z] [2026-10-04T09:27:05.810Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-10-04T09:27:05.810Z] ] *** [2026-10-04T09:27:06.155Z] skipping: [an-a01n01] [2026-10-04T09:27:06.155Z] skipping: [an-a01n02] [2026-10-04T09:27:06.155Z] [2026-10-04T09:27:06.155Z] PLAY [Copy iso images to striker01] ******************************************** [2026-10-04T09:27:06.155Z] [2026-10-04T09:27:06.155Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:27:07.876Z] ok: [an-striker01] [2026-10-04T09:27:07.876Z] [2026-10-04T09:27:07.876Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-10-04T09:27:08.615Z] ok: [an-striker01] [2026-10-04T09:27:08.615Z] [2026-10-04T09:27:08.615Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-10-04T09:27:09.810Z] changed: [an-striker01] [2026-10-04T09:27:09.810Z] [2026-10-04T09:27:09.810Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-10-04T09:27:17.234Z] changed: [an-striker01] => (item=deploy1.iso) [2026-10-04T09:27:17.234Z] changed: [an-striker01] => (item=deploy2.iso) [2026-10-04T09:27:17.234Z] changed: [an-striker01] => (item=deploy3.iso) [2026-10-04T09:27:17.234Z] changed: [an-striker01] => (item=deploy4.iso) [2026-10-04T09:27:17.234Z] changed: [an-striker01] => (item=deploy5.iso) [2026-10-04T09:27:17.234Z] [2026-10-04T09:27:17.234Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-10-04T09:27:17.234Z] ] *** [2026-10-04T09:27:17.974Z] changed: [an-striker01] [2026-10-04T09:27:17.974Z] [2026-10-04T09:27:17.974Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-10-04T09:27:19.169Z] changed: [an-striker01] [2026-10-04T09:27:19.169Z] [2026-10-04T09:27:19.169Z] PLAY [Check iso images propagation] ******************************************** [2026-10-04T09:27:19.169Z] [2026-10-04T09:27:19.169Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:27:20.890Z] ok: [an-a01n01] [2026-10-04T09:27:20.890Z] ok: [an-a01n02] [2026-10-04T09:27:20.890Z] ok: [an-striker01] [2026-10-04T09:27:20.890Z] ok: [an-striker02] [2026-10-04T09:27:20.890Z] [2026-10-04T09:27:20.890Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-10-04T09:27:20.890Z] exit 1 [2026-10-04T09:27:20.890Z] else [2026-10-04T09:27:20.890Z] exit 0 [2026-10-04T09:27:20.890Z] fi [2026-10-04T09:27:20.890Z] ] *** [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-10-04T09:28:59.399Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] ok: [an-striker02] [2026-10-04T09:28:59.399Z] ok: [an-a01n01] [2026-10-04T09:28:59.399Z] ok: [an-a01n02] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] PLAY [example tests] *********************************************************** [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:28:59.399Z] ok: [an-a01n01] [2026-10-04T09:28:59.399Z] ok: [an-a01n02] [2026-10-04T09:28:59.399Z] ok: [an-striker02] [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-10-04T09:28:59.399Z] ] *** [2026-10-04T09:28:59.399Z] skipping: [an-a01n01] [2026-10-04T09:28:59.399Z] skipping: [an-a01n02] [2026-10-04T09:28:59.399Z] ok: [an-striker02] [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-10-04T09:28:59.399Z] ] *** [2026-10-04T09:28:59.399Z] skipping: [an-striker01] [2026-10-04T09:28:59.399Z] skipping: [an-striker02] [2026-10-04T09:28:59.399Z] ok: [an-a01n01] [2026-10-04T09:28:59.399Z] ok: [an-a01n02] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [inject failure _raw_params=/usr/bin/false [2026-10-04T09:28:59.399Z] ] ******************************* [2026-10-04T09:28:59.399Z] skipping: [an-striker01] [2026-10-04T09:28:59.399Z] skipping: [an-striker02] [2026-10-04T09:28:59.399Z] skipping: [an-a01n01] [2026-10-04T09:28:59.399Z] skipping: [an-a01n02] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] PLAY [Record anvil available resources from striker01] ************************* [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-10-04T09:28:59.399Z] ] *** [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] PLAY [check selinux status] **************************************************** [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:28:59.399Z] ok: [an-a01n01] [2026-10-04T09:28:59.399Z] ok: [an-striker02] [2026-10-04T09:28:59.399Z] ok: [an-a01n02] [2026-10-04T09:28:59.399Z] ok: [an-striker01] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-10-04T09:28:59.399Z] skipping: [an-striker01] [2026-10-04T09:28:59.399Z] skipping: [an-striker02] [2026-10-04T09:28:59.399Z] skipping: [an-a01n01] [2026-10-04T09:28:59.399Z] skipping: [an-a01n02] [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-10-04T09:28:59.399Z] ok: [an-striker01] => { [2026-10-04T09:28:59.399Z] "ansible_selinux": { [2026-10-04T09:28:59.399Z] "config_mode": "enforcing", [2026-10-04T09:28:59.399Z] "mode": "enforcing", [2026-10-04T09:28:59.399Z] "policyvers": 33, [2026-10-04T09:28:59.399Z] "status": "enabled", [2026-10-04T09:28:59.399Z] "type": "targeted" [2026-10-04T09:28:59.399Z] }, [2026-10-04T09:28:59.399Z] "failed_when_result": false [2026-10-04T09:28:59.399Z] } [2026-10-04T09:28:59.399Z] ok: [an-striker02] => { [2026-10-04T09:28:59.399Z] "ansible_selinux": { [2026-10-04T09:28:59.399Z] "config_mode": "enforcing", [2026-10-04T09:28:59.399Z] "mode": "enforcing", [2026-10-04T09:28:59.399Z] "policyvers": 33, [2026-10-04T09:28:59.399Z] "status": "enabled", [2026-10-04T09:28:59.399Z] "type": "targeted" [2026-10-04T09:28:59.399Z] }, [2026-10-04T09:28:59.399Z] "failed_when_result": false [2026-10-04T09:28:59.399Z] } [2026-10-04T09:28:59.399Z] ok: [an-a01n01] => { [2026-10-04T09:28:59.399Z] "ansible_selinux": { [2026-10-04T09:28:59.399Z] "config_mode": "enforcing", [2026-10-04T09:28:59.399Z] "mode": "enforcing", [2026-10-04T09:28:59.399Z] "policyvers": 33, [2026-10-04T09:28:59.399Z] "status": "enabled", [2026-10-04T09:28:59.399Z] "type": "targeted" [2026-10-04T09:28:59.399Z] }, [2026-10-04T09:28:59.399Z] "failed_when_result": false [2026-10-04T09:28:59.399Z] } [2026-10-04T09:28:59.399Z] ok: [an-a01n02] => { [2026-10-04T09:28:59.399Z] "ansible_selinux": { [2026-10-04T09:28:59.399Z] "config_mode": "enforcing", [2026-10-04T09:28:59.399Z] "mode": "enforcing", [2026-10-04T09:28:59.399Z] "policyvers": 33, [2026-10-04T09:28:59.399Z] "status": "enabled", [2026-10-04T09:28:59.399Z] "type": "targeted" [2026-10-04T09:28:59.399Z] }, [2026-10-04T09:28:59.399Z] "failed_when_result": false [2026-10-04T09:28:59.399Z] } [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] PLAY [Create random file on striker01] ***************************************** [2026-10-04T09:28:59.399Z] [2026-10-04T09:28:59.399Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:29:01.122Z] ok: [an-striker01] [2026-10-04T09:29:01.122Z] [2026-10-04T09:29:01.122Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-10-04T09:29:02.316Z] changed: [an-striker01] [2026-10-04T09:29:02.316Z] [2026-10-04T09:29:02.316Z] PLAY [Check archive file propagation] ****************************************** [2026-10-04T09:29:02.316Z] [2026-10-04T09:29:02.316Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:29:04.654Z] ok: [an-a01n01] [2026-10-04T09:29:04.654Z] ok: [an-a01n02] [2026-10-04T09:29:04.654Z] ok: [an-striker01] [2026-10-04T09:29:04.654Z] ok: [an-striker02] [2026-10-04T09:29:04.654Z] [2026-10-04T09:29:04.654Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-10-04T09:29:04.654Z] exit 1 [2026-10-04T09:29:04.654Z] else [2026-10-04T09:29:04.654Z] exit 0 [2026-10-04T09:29:04.654Z] fi [2026-10-04T09:29:04.654Z] ] *** [2026-10-04T09:29:17.985Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-10-04T09:29:17.985Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-10-04T09:29:17.985Z] ok: [an-striker01] [2026-10-04T09:29:17.985Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-10-04T09:29:17.985Z] ok: [an-a01n01] [2026-10-04T09:29:17.985Z] ok: [an-a01n02] [2026-10-04T09:29:17.985Z] ok: [an-striker02] [2026-10-04T09:29:17.985Z] [2026-10-04T09:29:17.985Z] PLAY [test fencing] ************************************************************ [2026-10-04T09:29:17.985Z] [2026-10-04T09:29:17.985Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:29:17.985Z] ok: [an-a01n01] [2026-10-04T09:29:17.985Z] ok: [an-a01n02] [2026-10-04T09:29:17.985Z] [2026-10-04T09:29:17.985Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T09:29:17.985Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T09:29:17.985Z] [2026-10-04T09:29:17.985Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T09:29:17.985Z] ] *** [2026-10-04T09:29:21.876Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T09:29:21.876Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T09:29:21.876Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T09:29:21.876Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T09:29:21.876Z] [2026-10-04T09:29:21.876Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-10-04T09:29:21.876Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T09:29:21.876Z] [2026-10-04T09:29:21.876Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T09:29:21.876Z] ] ********** [2026-10-04T09:29:22.220Z] skipping: [an-a01n02] [2026-10-04T09:29:22.220Z] changed: [an-a01n01] [2026-10-04T09:29:22.220Z] [2026-10-04T09:29:22.220Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T09:29:22.220Z] skipping: [an-a01n02] [2026-10-04T09:29:22.220Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-10-04T09:29:22.220Z] [2026-10-04T09:29:22.220Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T09:32:46.422Z] ok: [an-a01n01 -> localhost] [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T09:32:46.422Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T09:32:46.422Z] ] *** [2026-10-04T09:32:46.422Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T09:32:46.422Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T09:32:46.422Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T09:32:46.422Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-10-04T09:32:46.422Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T09:32:46.422Z] ] ********** [2026-10-04T09:32:46.422Z] skipping: [an-a01n01] [2026-10-04T09:32:46.422Z] changed: [an-a01n02] [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T09:32:46.422Z] skipping: [an-a01n01] [2026-10-04T09:32:46.422Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-10-04T09:32:46.422Z] [2026-10-04T09:32:46.422Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T09:35:55.521Z] ok: [an-a01n02 -> localhost] [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T09:35:55.521Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T09:35:55.521Z] ] *** [2026-10-04T09:35:55.521Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T09:35:55.521Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T09:35:55.521Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T09:35:55.521Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] PLAY [advanced server tests] *************************************************** [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T09:35:55.521Z] ok: [an-a01n01] [2026-10-04T09:35:55.521Z] ok: [an-a01n02] [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T09:35:55.521Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T09:35:55.521Z] [2026-10-04T09:35:55.521Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T09:35:55.521Z] ] *** [2026-10-04T09:35:57.852Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T09:35:57.852Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T09:35:57.852Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T09:35:57.852Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T09:35:57.852Z] [2026-10-04T09:35:57.852Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-10-04T09:35:58.196Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-10-04T09:35:58.196Z] [2026-10-04T09:35:58.196Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-10-04T09:35:58.196Z] if [ -z "$storage" ]; then [2026-10-04T09:35:58.196Z] echo "Unable to detect shared storage" [2026-10-04T09:35:58.196Z] exit 1 [2026-10-04T09:35:58.196Z] fi [2026-10-04T09:35:58.196Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T09:35:58.196Z] ] *** [2026-10-04T09:36:04.239Z] changed: [an-a01n01] [2026-10-04T09:36:04.239Z] changed: [an-a01n02] [2026-10-04T09:36:04.239Z] [2026-10-04T09:36:04.239Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-10-04T09:36:04.239Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-10-04T09:36:04.239Z] [2026-10-04T09:36:04.239Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-10-04T09:36:04.239Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-10-04T09:36:04.239Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-10-04T09:36:04.239Z] echo "Unable to detect all deploy iso" [2026-10-04T09:36:04.239Z] echo "Detected isos" [2026-10-04T09:36:04.239Z] echo "$isos" [2026-10-04T09:36:04.239Z] exit 1 [2026-10-04T09:36:04.239Z] fi [2026-10-04T09:36:04.239Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-10-04T09:36:04.239Z] if [ -z "$installiso" ]; then [2026-10-04T09:36:04.239Z] echo "Unable to detect install iso" [2026-10-04T09:36:04.239Z] echo "Detected isos" [2026-10-04T09:36:04.239Z] echo "$isos" [2026-10-04T09:36:04.239Z] exit 1 [2026-10-04T09:36:04.239Z] fi [2026-10-04T09:36:04.239Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T09:36:04.239Z] ] *** [2026-10-04T09:36:09.145Z] changed: [an-a01n02] [2026-10-04T09:36:09.145Z] changed: [an-a01n01] [2026-10-04T09:36:09.145Z] [2026-10-04T09:36:09.145Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-10-04T09:36:09.145Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T09:36:09.145Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T09:36:09.145Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T09:36:09.145Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T09:36:09.145Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T09:36:09.145Z] [2026-10-04T09:36:09.145Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T09:36:09.145Z] ] *** [2026-10-04T09:36:10.864Z] skipping: [an-a01n02] [2026-10-04T09:36:10.864Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.826166", "end": "2026-10-04 05:36:10.402446", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 05:36:09.576280", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T09:36:10.864Z] ...ignoring [2026-10-04T09:36:10.864Z] [2026-10-04T09:36:10.864Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T09:36:10.864Z] --ci-test \ [2026-10-04T09:36:10.864Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T09:36:10.864Z] --os almalinux9 \ [2026-10-04T09:36:10.864Z] --cpu 2 --ram 4G \ [2026-10-04T09:36:10.864Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T09:36:10.864Z] --install-media {{ installisouuid }} \ [2026-10-04T09:36:10.864Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T09:36:10.864Z] ] *** [2026-10-04T09:36:19.920Z] skipping: [an-a01n02] [2026-10-04T09:36:19.920Z] changed: [an-a01n01] [2026-10-04T09:36:19.920Z] [2026-10-04T09:36:19.920Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T09:36:19.920Z] skipping: [an-a01n01] [2026-10-04T09:36:19.920Z] skipping: [an-a01n02] [2026-10-04T09:36:19.920Z] [2026-10-04T09:36:19.920Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T09:36:19.920Z] skipping: [an-a01n01] [2026-10-04T09:36:19.920Z] skipping: [an-a01n02] [2026-10-04T09:36:19.920Z] [2026-10-04T09:36:19.920Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T09:36:19.920Z] skipping: [an-a01n01] [2026-10-04T09:36:19.920Z] skipping: [an-a01n02] [2026-10-04T09:36:19.920Z] [2026-10-04T09:36:19.920Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T09:36:19.920Z] skipping: [an-a01n01] [2026-10-04T09:36:19.921Z] skipping: [an-a01n02] [2026-10-04T09:36:19.921Z] [2026-10-04T09:36:19.921Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T09:36:19.921Z] ] *** [2026-10-04T09:36:21.113Z] skipping: [an-a01n02] [2026-10-04T09:36:21.113Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.968242", "end": "2026-10-04 05:36:20.774806", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 05:36:19.806564", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T09:36:21.113Z] ...ignoring [2026-10-04T09:36:21.113Z] [2026-10-04T09:36:21.113Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T09:36:21.113Z] --ci-test \ [2026-10-04T09:36:21.113Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T09:36:21.113Z] --os almalinux9 \ [2026-10-04T09:36:21.113Z] --cpu 2 --ram 4G \ [2026-10-04T09:36:21.113Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T09:36:21.113Z] --install-media {{ installisouuid }} \ [2026-10-04T09:36:21.113Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T09:36:21.113Z] ] *** [2026-10-04T09:36:30.577Z] skipping: [an-a01n02] [2026-10-04T09:36:30.577Z] changed: [an-a01n01] [2026-10-04T09:36:30.577Z] [2026-10-04T09:36:30.577Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T09:36:30.577Z] skipping: [an-a01n01] [2026-10-04T09:36:30.577Z] skipping: [an-a01n02] [2026-10-04T09:36:30.577Z] [2026-10-04T09:36:30.578Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T09:36:30.578Z] skipping: [an-a01n01] [2026-10-04T09:36:30.578Z] skipping: [an-a01n02] [2026-10-04T09:36:30.578Z] [2026-10-04T09:36:30.578Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T09:36:30.578Z] skipping: [an-a01n01] [2026-10-04T09:36:30.578Z] skipping: [an-a01n02] [2026-10-04T09:36:30.578Z] [2026-10-04T09:36:30.578Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T09:36:30.578Z] skipping: [an-a01n01] [2026-10-04T09:36:30.578Z] skipping: [an-a01n02] [2026-10-04T09:36:30.578Z] [2026-10-04T09:36:30.578Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T09:36:30.578Z] ] *** [2026-10-04T09:36:30.921Z] skipping: [an-a01n02] [2026-10-04T09:36:30.921Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.886983", "end": "2026-10-04 05:36:30.759293", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 05:36:29.872310", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T09:36:30.921Z] ...ignoring [2026-10-04T09:36:30.921Z] [2026-10-04T09:36:30.921Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T09:36:30.921Z] --ci-test \ [2026-10-04T09:36:30.921Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T09:36:30.921Z] --os almalinux9 \ [2026-10-04T09:36:30.921Z] --cpu 2 --ram 4G \ [2026-10-04T09:36:30.921Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T09:36:30.921Z] --install-media {{ installisouuid }} \ [2026-10-04T09:36:30.921Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T09:36:30.921Z] ] *** [2026-10-04T09:36:39.984Z] skipping: [an-a01n02] [2026-10-04T09:36:39.984Z] changed: [an-a01n01] [2026-10-04T09:36:39.984Z] [2026-10-04T09:36:39.984Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T09:36:39.984Z] skipping: [an-a01n01] [2026-10-04T09:36:39.984Z] skipping: [an-a01n02] [2026-10-04T09:36:39.984Z] [2026-10-04T09:36:39.984Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T09:36:39.984Z] skipping: [an-a01n01] [2026-10-04T09:36:39.984Z] skipping: [an-a01n02] [2026-10-04T09:36:39.984Z] [2026-10-04T09:36:39.984Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T09:36:39.984Z] skipping: [an-a01n01] [2026-10-04T09:36:39.984Z] skipping: [an-a01n02] [2026-10-04T09:36:39.984Z] [2026-10-04T09:36:39.984Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T09:36:39.984Z] skipping: [an-a01n01] [2026-10-04T09:36:39.984Z] skipping: [an-a01n02] [2026-10-04T09:36:39.984Z] [2026-10-04T09:36:39.984Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T09:36:39.984Z] ] *** [2026-10-04T09:36:40.722Z] skipping: [an-a01n02] [2026-10-04T09:36:40.722Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.966216", "end": "2026-10-04 05:36:40.578496", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 05:36:39.612280", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T09:36:40.722Z] ...ignoring [2026-10-04T09:36:40.722Z] [2026-10-04T09:36:40.722Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T09:36:40.722Z] --ci-test \ [2026-10-04T09:36:40.722Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T09:36:40.722Z] --os almalinux9 \ [2026-10-04T09:36:40.722Z] --cpu 2 --ram 4G \ [2026-10-04T09:36:40.722Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T09:36:40.722Z] --install-media {{ installisouuid }} \ [2026-10-04T09:36:40.722Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T09:36:40.722Z] ] *** [2026-10-04T09:36:49.776Z] skipping: [an-a01n02] [2026-10-04T09:36:49.776Z] changed: [an-a01n01] [2026-10-04T09:36:49.776Z] [2026-10-04T09:36:49.776Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T09:36:49.776Z] skipping: [an-a01n01] [2026-10-04T09:36:49.776Z] skipping: [an-a01n02] [2026-10-04T09:36:49.776Z] [2026-10-04T09:36:49.776Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T09:36:49.776Z] skipping: [an-a01n01] [2026-10-04T09:36:49.776Z] skipping: [an-a01n02] [2026-10-04T09:36:49.776Z] [2026-10-04T09:36:49.776Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T09:36:49.776Z] skipping: [an-a01n01] [2026-10-04T09:36:49.776Z] skipping: [an-a01n02] [2026-10-04T09:36:49.776Z] [2026-10-04T09:36:49.776Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T09:36:49.776Z] skipping: [an-a01n01] [2026-10-04T09:36:49.776Z] skipping: [an-a01n02] [2026-10-04T09:36:49.776Z] [2026-10-04T09:36:49.776Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T09:36:49.776Z] ] *** [2026-10-04T09:36:50.967Z] skipping: [an-a01n02] [2026-10-04T09:36:50.967Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.947155", "end": "2026-10-04 05:36:50.678896", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 05:36:49.731741", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T09:36:50.967Z] ...ignoring [2026-10-04T09:36:50.967Z] [2026-10-04T09:36:50.967Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T09:36:50.967Z] --ci-test \ [2026-10-04T09:36:50.967Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T09:36:50.967Z] --os almalinux9 \ [2026-10-04T09:36:50.967Z] --cpu 2 --ram 4G \ [2026-10-04T09:36:50.967Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T09:36:50.967Z] --install-media {{ installisouuid }} \ [2026-10-04T09:36:50.967Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T09:36:50.967Z] ] *** [2026-10-04T09:37:00.021Z] skipping: [an-a01n02] [2026-10-04T09:37:00.021Z] changed: [an-a01n01] [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T09:37:00.021Z] skipping: [an-a01n01] [2026-10-04T09:37:00.021Z] skipping: [an-a01n02] [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T09:37:00.021Z] skipping: [an-a01n01] [2026-10-04T09:37:00.021Z] skipping: [an-a01n02] [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T09:37:00.021Z] skipping: [an-a01n01] [2026-10-04T09:37:00.021Z] skipping: [an-a01n02] [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T09:37:00.021Z] skipping: [an-a01n01] [2026-10-04T09:37:00.021Z] skipping: [an-a01n02] [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-10-04T09:37:00.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T09:37:00.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T09:37:00.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T09:37:00.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T09:37:00.021Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T09:37:00.021Z] [2026-10-04T09:37:00.021Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T09:37:00.021Z] ] *** [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (358 retries left). [2026-10-04T09:37:40.376Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (358 retries left). [2026-10-04T09:37:40.376Z] ok: [an-a01n02] [2026-10-04T09:37:40.376Z] ok: [an-a01n01] [2026-10-04T09:37:40.376Z] [2026-10-04T09:37:40.376Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T09:37:40.376Z] ] *** [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-10-04T09:39:49.068Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-10-04T09:39:49.069Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-10-04T09:39:49.069Z] ok: [an-a01n02] [2026-10-04T09:39:49.069Z] ok: [an-a01n01] [2026-10-04T09:39:49.069Z] [2026-10-04T09:39:49.069Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T09:39:49.069Z] ] *** [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-10-04T09:41:57.786Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-10-04T09:41:57.786Z] ok: [an-a01n02] [2026-10-04T09:41:57.786Z] ok: [an-a01n01] [2026-10-04T09:41:57.786Z] [2026-10-04T09:41:57.786Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T09:41:57.786Z] ] *** [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-10-04T09:44:06.501Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-10-04T09:44:06.501Z] ok: [an-a01n02] [2026-10-04T09:44:06.501Z] ok: [an-a01n01] [2026-10-04T09:44:06.501Z] [2026-10-04T09:44:06.501Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T09:44:06.501Z] ] *** [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-10-04T09:47:15.608Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-10-04T09:47:15.608Z] ok: [an-a01n02] [2026-10-04T09:47:15.608Z] ok: [an-a01n01] [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-10-04T09:47:15.608Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T09:47:15.608Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T09:47:15.608Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T09:47:15.608Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T09:47:15.608Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T09:47:15.608Z] ] *** [2026-10-04T09:47:15.608Z] ok: [an-a01n02] [2026-10-04T09:47:15.608Z] ok: [an-a01n01] [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T09:47:15.608Z] ] *** [2026-10-04T09:47:15.608Z] ok: [an-a01n02] [2026-10-04T09:47:15.608Z] ok: [an-a01n01] [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T09:47:15.608Z] ] *** [2026-10-04T09:47:15.608Z] ok: [an-a01n02] [2026-10-04T09:47:15.608Z] ok: [an-a01n01] [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T09:47:15.608Z] ] *** [2026-10-04T09:47:15.608Z] ok: [an-a01n02] [2026-10-04T09:47:15.608Z] ok: [an-a01n01] [2026-10-04T09:47:15.608Z] [2026-10-04T09:47:15.608Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T09:47:15.608Z] ] *** [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-10-04T09:48:13.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-10-04T09:48:13.640Z] ok: [an-a01n02] [2026-10-04T09:48:13.640Z] ok: [an-a01n01] [2026-10-04T09:48:13.640Z] [2026-10-04T09:48:13.640Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-10-04T09:48:13.640Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=1) [2026-10-04T09:48:13.640Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=2) [2026-10-04T09:48:13.640Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=3) [2026-10-04T09:48:13.640Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=4) [2026-10-04T09:48:13.640Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n02, an-a01n01 => (item=5) [2026-10-04T09:48:13.640Z] [2026-10-04T09:48:13.640Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T09:48:13.640Z] ] *** [2026-10-04T09:48:13.640Z] ok: [an-a01n02] [2026-10-04T09:48:13.640Z] ok: [an-a01n01] [2026-10-04T09:48:13.640Z] [2026-10-04T09:48:13.640Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T09:48:13.640Z] ] *** [2026-10-04T09:48:13.640Z] ok: [an-a01n02] [2026-10-04T09:48:13.640Z] ok: [an-a01n01] [2026-10-04T09:48:13.640Z] [2026-10-04T09:48:13.640Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T09:48:13.640Z] ] *** [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-10-04T09:50:07.242Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-10-04T09:50:07.242Z] ok: [an-a01n02] [2026-10-04T09:50:07.242Z] ok: [an-a01n01] [2026-10-04T09:50:07.242Z] [2026-10-04T09:50:07.242Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T09:50:07.242Z] ] *** [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-10-04T09:52:15.963Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-10-04T09:52:15.963Z] ok: [an-a01n02] [2026-10-04T09:52:15.963Z] ok: [an-a01n01] [2026-10-04T09:52:15.963Z] [2026-10-04T09:52:15.963Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T09:52:15.963Z] ] *** [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-10-04T09:55:09.948Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-10-04T09:55:09.948Z] ok: [an-a01n02] [2026-10-04T09:55:09.948Z] ok: [an-a01n01] [2026-10-04T09:55:09.948Z] [2026-10-04T09:55:09.948Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-10-04T09:55:09.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T09:55:09.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T09:55:09.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T09:55:09.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T09:55:09.948Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T09:55:09.948Z] [2026-10-04T09:55:09.948Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T09:55:09.948Z] ] *** [2026-10-04T09:55:10.687Z] ok: [an-a01n02] [2026-10-04T09:55:10.687Z] ok: [an-a01n01] [2026-10-04T09:55:10.687Z] [2026-10-04T09:55:10.687Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T09:55:10.687Z] ] *** [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (352 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (352 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (351 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (351 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (350 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (350 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (349 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (348 retries left). [2026-10-04T09:57:49.597Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (349 retries left). [2026-10-04T09:57:49.597Z] ok: [an-a01n02] [2026-10-04T09:57:49.597Z] ok: [an-a01n01] [2026-10-04T09:57:49.597Z] [2026-10-04T09:57:49.597Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T09:57:49.597Z] ] *** [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-10-04T10:00:28.478Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-10-04T10:00:28.478Z] ok: [an-a01n02] [2026-10-04T10:00:28.478Z] ok: [an-a01n01] [2026-10-04T10:00:28.478Z] [2026-10-04T10:00:28.478Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:00:28.478Z] ] *** [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-10-04T10:01:51.912Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-10-04T10:01:51.912Z] ok: [an-a01n01] [2026-10-04T10:01:51.912Z] ok: [an-a01n02] [2026-10-04T10:01:51.912Z] [2026-10-04T10:01:51.912Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:01:51.912Z] ] *** [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-10-04T10:03:45.504Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-10-04T10:03:45.504Z] ok: [an-a01n01] [2026-10-04T10:03:45.504Z] ok: [an-a01n02] [2026-10-04T10:03:45.504Z] [2026-10-04T10:03:45.504Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:03:45.504Z] [2026-10-04T10:03:45.504Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-10-04T10:03:45.504Z] ok: [an-a01n01] [2026-10-04T10:03:45.504Z] ok: [an-a01n02] [2026-10-04T10:03:45.504Z] [2026-10-04T10:03:45.504Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:03:45.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:03:45.504Z] [2026-10-04T10:03:45.504Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:03:45.504Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:03:45.504Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:03:45.504Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:03:45.504Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:03:45.504Z] echo "pcs:" [2026-10-04T10:03:45.504Z] echo "$pcsstatus" [2026-10-04T10:03:45.504Z] exit 1 [2026-10-04T10:03:45.504Z] fi [2026-10-04T10:03:45.504Z] exit 0 [2026-10-04T10:03:45.504Z] ] *** [2026-10-04T10:03:45.504Z] ok: [an-a01n02] [2026-10-04T10:03:45.504Z] ok: [an-a01n01] [2026-10-04T10:03:45.504Z] [2026-10-04T10:03:45.504Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:03:45.504Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:03:45.504Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:03:45.504Z] ret=0 [2026-10-04T10:03:45.504Z] if [ "$servers" != "1" ]; then [2026-10-04T10:03:45.504Z] echo "Not all servers are running on the same node!" [2026-10-04T10:03:45.504Z] echo "$pcsstatus" [2026-10-04T10:03:45.504Z] ret=1 [2026-10-04T10:03:45.504Z] fi [2026-10-04T10:03:45.504Z] echo "$hypervisor" [2026-10-04T10:03:45.504Z] exit $ret [2026-10-04T10:03:45.504Z] ] *** [2026-10-04T10:03:46.697Z] ok: [an-a01n02] [2026-10-04T10:03:46.697Z] ok: [an-a01n01] [2026-10-04T10:03:46.697Z] [2026-10-04T10:03:46.697Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-10-04T10:03:46.697Z] ok: [an-a01n01] [2026-10-04T10:03:46.697Z] ok: [an-a01n02] [2026-10-04T10:03:46.697Z] [2026-10-04T10:03:46.697Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-10-04T10:03:46.697Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:03:46.697Z] [2026-10-04T10:03:46.697Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T10:03:46.697Z] ] ********** [2026-10-04T10:03:47.435Z] skipping: [an-a01n02] [2026-10-04T10:03:47.435Z] changed: [an-a01n01] [2026-10-04T10:03:47.435Z] [2026-10-04T10:03:47.435Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T10:03:47.435Z] skipping: [an-a01n01] [2026-10-04T10:03:47.435Z] skipping: [an-a01n02] [2026-10-04T10:03:47.435Z] [2026-10-04T10:03:47.435Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-10-04T10:03:47.435Z] skipping: [an-a01n01] [2026-10-04T10:03:47.435Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-10-04T10:03:47.435Z] [2026-10-04T10:03:47.435Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-10-04T10:03:47.435Z] ] *** [2026-10-04T10:04:57.004Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-10-04T10:04:57.004Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-10-04T10:04:57.004Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-10-04T10:04:57.004Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-10-04T10:04:57.004Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-10-04T10:04:57.004Z] ok: [an-a01n02] [2026-10-04T10:04:57.004Z] [2026-10-04T10:04:57.004Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:04:57.004Z] skipping: [an-a01n01] [2026-10-04T10:04:57.004Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-10-04T10:04:57.004Z] [2026-10-04T10:04:57.004Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:04:57.004Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:04:57.004Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:04:57.004Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:04:57.004Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:04:57.004Z] echo "pcs:" [2026-10-04T10:04:57.004Z] echo "$pcsstatus" [2026-10-04T10:04:57.004Z] exit 1 [2026-10-04T10:04:57.004Z] fi [2026-10-04T10:04:57.004Z] exit 0 [2026-10-04T10:04:57.004Z] ] *** [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-10-04T10:06:35.607Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:06:35.607Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:06:35.607Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:06:35.607Z] ret=0 [2026-10-04T10:06:35.607Z] if [ "$servers" != "1" ]; then [2026-10-04T10:06:35.607Z] echo "Not all servers are running on the same node!" [2026-10-04T10:06:35.607Z] echo "$pcsstatus" [2026-10-04T10:06:35.607Z] ret=1 [2026-10-04T10:06:35.607Z] fi [2026-10-04T10:06:35.607Z] echo "$hypervisor" [2026-10-04T10:06:35.607Z] exit $ret [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] => (item=1) [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] => (item=2) [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] => (item=3) [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] => (item=4) [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] => (item=5) [2026-10-04T10:06:35.607Z] skipping: [an-a01n01] [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:06:35.607Z] ] *** [2026-10-04T10:06:35.607Z] ok: [an-a01n02] [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-10-04T10:06:35.607Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:06:35.607Z] [2026-10-04T10:06:35.607Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T10:09:59.783Z] ok: [an-a01n02 -> localhost] [2026-10-04T10:09:59.783Z] ok: [an-a01n01 -> localhost] [2026-10-04T10:09:59.783Z] [2026-10-04T10:09:59.783Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:09:59.783Z] [2026-10-04T10:09:59.783Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:09:59.783Z] ] *** [2026-10-04T10:09:59.783Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:09:59.783Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:09:59.783Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:09:59.783Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:09:59.783Z] [2026-10-04T10:09:59.783Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:09:59.783Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:09:59.783Z] [2026-10-04T10:09:59.783Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:09:59.783Z] ] *** [2026-10-04T10:11:23.202Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-10-04T10:11:23.202Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-10-04T10:11:23.202Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-10-04T10:11:23.202Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-10-04T10:11:23.203Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:11:23.203Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:11:23.203Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:11:23.203Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:11:23.203Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:11:23.203Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:11:23.203Z] echo "pcs:" [2026-10-04T10:11:23.203Z] echo "$pcsstatus" [2026-10-04T10:11:23.203Z] exit 1 [2026-10-04T10:11:23.203Z] fi [2026-10-04T10:11:23.203Z] exit 0 [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:11:23.203Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:11:23.203Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:11:23.203Z] ret=0 [2026-10-04T10:11:23.203Z] if [ "$servers" != "1" ]; then [2026-10-04T10:11:23.203Z] echo "Not all servers are running on the same node!" [2026-10-04T10:11:23.203Z] echo "$pcsstatus" [2026-10-04T10:11:23.203Z] ret=1 [2026-10-04T10:11:23.203Z] fi [2026-10-04T10:11:23.203Z] echo "$hypervisor" [2026-10-04T10:11:23.203Z] exit $ret [2026-10-04T10:11:23.203Z] ] *** [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-10-04T10:11:23.203Z] ok: [an-a01n01] [2026-10-04T10:11:23.203Z] ok: [an-a01n02] [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-10-04T10:11:23.203Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:11:23.203Z] [2026-10-04T10:11:23.203Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T10:11:23.203Z] ] ********** [2026-10-04T10:11:23.571Z] skipping: [an-a01n01] [2026-10-04T10:11:23.571Z] changed: [an-a01n02] [2026-10-04T10:11:23.571Z] [2026-10-04T10:11:23.571Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T10:11:23.571Z] skipping: [an-a01n01] [2026-10-04T10:11:23.571Z] skipping: [an-a01n02] [2026-10-04T10:11:23.571Z] [2026-10-04T10:11:23.571Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-10-04T10:11:23.571Z] skipping: [an-a01n02] [2026-10-04T10:11:23.571Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-10-04T10:11:23.571Z] [2026-10-04T10:11:23.571Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-10-04T10:11:23.571Z] ] *** [2026-10-04T10:12:33.136Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-10-04T10:12:33.136Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-10-04T10:12:33.136Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-10-04T10:12:33.136Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-10-04T10:12:33.136Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-10-04T10:12:33.136Z] ok: [an-a01n01] [2026-10-04T10:12:33.136Z] [2026-10-04T10:12:33.136Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:12:33.136Z] skipping: [an-a01n02] [2026-10-04T10:12:33.136Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-10-04T10:12:33.136Z] [2026-10-04T10:12:33.136Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:12:33.136Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:12:33.136Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:12:33.136Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:12:33.136Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:12:33.136Z] echo "pcs:" [2026-10-04T10:12:33.136Z] echo "$pcsstatus" [2026-10-04T10:12:33.136Z] exit 1 [2026-10-04T10:12:33.136Z] fi [2026-10-04T10:12:33.136Z] exit 0 [2026-10-04T10:12:33.136Z] ] *** [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-10-04T10:13:42.714Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-10-04T10:13:42.714Z] ok: [an-a01n01] [2026-10-04T10:13:42.714Z] [2026-10-04T10:13:42.714Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:13:42.714Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:13:42.714Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:13:42.714Z] ret=0 [2026-10-04T10:13:42.714Z] if [ "$servers" != "1" ]; then [2026-10-04T10:13:42.714Z] echo "Not all servers are running on the same node!" [2026-10-04T10:13:42.714Z] echo "$pcsstatus" [2026-10-04T10:13:42.714Z] ret=1 [2026-10-04T10:13:42.714Z] fi [2026-10-04T10:13:42.714Z] echo "$hypervisor" [2026-10-04T10:13:42.714Z] exit $ret [2026-10-04T10:13:42.714Z] ] *** [2026-10-04T10:13:42.714Z] ok: [an-a01n01] [2026-10-04T10:13:42.714Z] [2026-10-04T10:13:42.714Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] => (item=1) [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] => (item=2) [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] => (item=3) [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] => (item=4) [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] => (item=5) [2026-10-04T10:13:42.714Z] skipping: [an-a01n02] [2026-10-04T10:13:42.714Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-10-04T10:13:42.714Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-10-04T10:13:42.714Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-10-04T10:13:42.714Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-10-04T10:13:42.714Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-10-04T10:13:42.714Z] [2026-10-04T10:13:42.714Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:13:42.714Z] ] *** [2026-10-04T10:13:45.052Z] ok: [an-a01n01] [2026-10-04T10:13:45.052Z] [2026-10-04T10:13:45.052Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:13:45.052Z] ] *** [2026-10-04T10:14:01.133Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-10-04T10:14:01.133Z] ok: [an-a01n01] [2026-10-04T10:14:01.133Z] [2026-10-04T10:14:01.133Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:14:01.133Z] ] *** [2026-10-04T10:14:01.133Z] ok: [an-a01n01] [2026-10-04T10:14:01.133Z] [2026-10-04T10:14:01.133Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:14:01.133Z] ] *** [2026-10-04T10:14:01.870Z] ok: [an-a01n01] [2026-10-04T10:14:01.870Z] [2026-10-04T10:14:01.870Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:14:01.870Z] ] *** [2026-10-04T10:14:02.231Z] ok: [an-a01n01] [2026-10-04T10:14:02.231Z] [2026-10-04T10:14:02.231Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-10-04T10:14:02.575Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:14:02.575Z] [2026-10-04T10:14:02.575Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T10:17:26.774Z] ok: [an-a01n01 -> localhost] [2026-10-04T10:17:26.774Z] ok: [an-a01n02 -> localhost] [2026-10-04T10:17:26.774Z] [2026-10-04T10:17:26.774Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:17:26.774Z] [2026-10-04T10:17:26.774Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:17:26.774Z] ] *** [2026-10-04T10:17:26.774Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:17:26.774Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:17:26.774Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:17:26.774Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:17:26.774Z] [2026-10-04T10:17:26.774Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:17:26.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:17:26.774Z] [2026-10-04T10:17:26.774Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:17:26.774Z] ] *** [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (337 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (337 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (336 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (336 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (335 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (335 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (334 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (334 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (333 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (333 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (332 retries left). [2026-10-04T10:22:36.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (332 retries left). [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:22:36.640Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:22:36.640Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:22:36.640Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:22:36.640Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:22:36.640Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:22:36.640Z] echo "pcs:" [2026-10-04T10:22:36.640Z] echo "$pcsstatus" [2026-10-04T10:22:36.640Z] exit 1 [2026-10-04T10:22:36.640Z] fi [2026-10-04T10:22:36.640Z] exit 0 [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:22:36.640Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:22:36.640Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:22:36.640Z] ret=0 [2026-10-04T10:22:36.640Z] if [ "$servers" != "1" ]; then [2026-10-04T10:22:36.640Z] echo "Not all servers are running on the same node!" [2026-10-04T10:22:36.640Z] echo "$pcsstatus" [2026-10-04T10:22:36.640Z] ret=1 [2026-10-04T10:22:36.640Z] fi [2026-10-04T10:22:36.640Z] echo "$hypervisor" [2026-10-04T10:22:36.640Z] exit $ret [2026-10-04T10:22:36.640Z] ] *** [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-10-04T10:22:36.640Z] ok: [an-a01n01] [2026-10-04T10:22:36.640Z] ok: [an-a01n02] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-10-04T10:22:36.640Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T10:22:36.640Z] ] ********** [2026-10-04T10:22:36.640Z] skipping: [an-a01n02] [2026-10-04T10:22:36.640Z] changed: [an-a01n01] [2026-10-04T10:22:36.640Z] [2026-10-04T10:22:36.640Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T10:22:36.984Z] skipping: [an-a01n01] [2026-10-04T10:22:36.984Z] skipping: [an-a01n02] [2026-10-04T10:22:36.984Z] [2026-10-04T10:22:36.984Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-10-04T10:22:36.984Z] skipping: [an-a01n01] [2026-10-04T10:22:36.984Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-10-04T10:22:36.984Z] [2026-10-04T10:22:36.984Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-10-04T10:22:36.984Z] ] *** [2026-10-04T10:23:46.553Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-10-04T10:23:46.553Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-10-04T10:23:46.553Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-10-04T10:23:46.553Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-10-04T10:23:46.553Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-10-04T10:23:46.553Z] ok: [an-a01n02] [2026-10-04T10:23:46.553Z] [2026-10-04T10:23:46.553Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:23:46.553Z] skipping: [an-a01n01] [2026-10-04T10:23:46.553Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-10-04T10:23:46.553Z] [2026-10-04T10:23:46.553Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:23:46.553Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:23:46.553Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:23:46.553Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:23:46.553Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:23:46.553Z] echo "pcs:" [2026-10-04T10:23:46.553Z] echo "$pcsstatus" [2026-10-04T10:23:46.553Z] exit 1 [2026-10-04T10:23:46.553Z] fi [2026-10-04T10:23:46.553Z] exit 0 [2026-10-04T10:23:46.553Z] ] *** [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-10-04T10:24:56.131Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-10-04T10:24:56.131Z] ok: [an-a01n02] [2026-10-04T10:24:56.131Z] [2026-10-04T10:24:56.131Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:24:56.131Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:24:56.131Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:24:56.131Z] ret=0 [2026-10-04T10:24:56.131Z] if [ "$servers" != "1" ]; then [2026-10-04T10:24:56.131Z] echo "Not all servers are running on the same node!" [2026-10-04T10:24:56.131Z] echo "$pcsstatus" [2026-10-04T10:24:56.131Z] ret=1 [2026-10-04T10:24:56.131Z] fi [2026-10-04T10:24:56.131Z] echo "$hypervisor" [2026-10-04T10:24:56.131Z] exit $ret [2026-10-04T10:24:56.131Z] ] *** [2026-10-04T10:24:56.131Z] ok: [an-a01n02] [2026-10-04T10:24:56.131Z] [2026-10-04T10:24:56.131Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] => (item=1) [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] => (item=2) [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] => (item=3) [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] => (item=4) [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] => (item=5) [2026-10-04T10:24:56.131Z] skipping: [an-a01n01] [2026-10-04T10:24:56.131Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-10-04T10:24:56.131Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-10-04T10:24:56.131Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-10-04T10:24:56.131Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-10-04T10:24:56.131Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-10-04T10:24:56.131Z] [2026-10-04T10:24:56.131Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:24:56.131Z] ] *** [2026-10-04T10:25:12.213Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-10-04T10:25:12.213Z] ok: [an-a01n02] [2026-10-04T10:25:12.213Z] [2026-10-04T10:25:12.213Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:25:12.213Z] ] *** [2026-10-04T10:25:12.213Z] ok: [an-a01n02] [2026-10-04T10:25:12.213Z] [2026-10-04T10:25:12.213Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:25:12.213Z] ] *** [2026-10-04T10:25:12.213Z] ok: [an-a01n02] [2026-10-04T10:25:12.213Z] [2026-10-04T10:25:12.213Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:25:12.213Z] ] *** [2026-10-04T10:25:12.558Z] ok: [an-a01n02] [2026-10-04T10:25:12.558Z] [2026-10-04T10:25:12.558Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:25:12.558Z] ] *** [2026-10-04T10:25:13.297Z] ok: [an-a01n02] [2026-10-04T10:25:13.297Z] [2026-10-04T10:25:13.297Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-10-04T10:25:13.297Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:25:13.297Z] [2026-10-04T10:25:13.297Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01 -> localhost] [2026-10-04T10:28:37.504Z] ok: [an-a01n02 -> localhost] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:28:37.504Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:28:37.504Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:28:37.504Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:28:37.504Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:28:37.504Z] [2026-10-04T10:28:37.504Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:28:37.504Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:28:37.504Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:28:37.504Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:28:37.504Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:28:37.504Z] echo "pcs:" [2026-10-04T10:28:37.504Z] echo "$pcsstatus" [2026-10-04T10:28:37.504Z] exit 1 [2026-10-04T10:28:37.504Z] fi [2026-10-04T10:28:37.504Z] exit 0 [2026-10-04T10:28:37.504Z] ] *** [2026-10-04T10:28:37.504Z] ok: [an-a01n01] [2026-10-04T10:28:37.504Z] ok: [an-a01n02] [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:28:37.505Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:28:37.505Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:28:37.505Z] ret=0 [2026-10-04T10:28:37.505Z] if [ "$servers" != "1" ]; then [2026-10-04T10:28:37.505Z] echo "Not all servers are running on the same node!" [2026-10-04T10:28:37.505Z] echo "$pcsstatus" [2026-10-04T10:28:37.505Z] ret=1 [2026-10-04T10:28:37.505Z] fi [2026-10-04T10:28:37.505Z] echo "$hypervisor" [2026-10-04T10:28:37.505Z] exit $ret [2026-10-04T10:28:37.505Z] ] *** [2026-10-04T10:28:37.505Z] ok: [an-a01n01] [2026-10-04T10:28:37.505Z] ok: [an-a01n02] [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-10-04T10:28:37.505Z] ok: [an-a01n01] [2026-10-04T10:28:37.505Z] ok: [an-a01n02] [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-10-04T10:28:37.505Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T10:28:37.505Z] ] ********** [2026-10-04T10:28:37.505Z] skipping: [an-a01n01] [2026-10-04T10:28:37.505Z] changed: [an-a01n02] [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T10:28:37.505Z] skipping: [an-a01n01] [2026-10-04T10:28:37.505Z] skipping: [an-a01n02] [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-10-04T10:28:37.505Z] skipping: [an-a01n02] [2026-10-04T10:28:37.505Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-10-04T10:28:37.505Z] [2026-10-04T10:28:37.505Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-10-04T10:28:37.505Z] ] *** [2026-10-04T10:29:47.072Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-10-04T10:29:47.072Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-10-04T10:29:47.072Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-10-04T10:29:47.072Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-10-04T10:29:47.072Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-10-04T10:29:47.072Z] ok: [an-a01n01] [2026-10-04T10:29:47.072Z] [2026-10-04T10:29:47.072Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:29:47.072Z] skipping: [an-a01n02] [2026-10-04T10:29:47.072Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-10-04T10:29:47.072Z] [2026-10-04T10:29:47.072Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:29:47.072Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:29:47.072Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:29:47.072Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:29:47.072Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:29:47.072Z] echo "pcs:" [2026-10-04T10:29:47.072Z] echo "$pcsstatus" [2026-10-04T10:29:47.072Z] exit 1 [2026-10-04T10:29:47.072Z] fi [2026-10-04T10:29:47.072Z] exit 0 [2026-10-04T10:29:47.072Z] ] *** [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-10-04T10:30:56.649Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-10-04T10:30:56.649Z] ok: [an-a01n01] [2026-10-04T10:30:56.649Z] [2026-10-04T10:30:56.649Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:30:56.649Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:30:56.649Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:30:56.649Z] ret=0 [2026-10-04T10:30:56.649Z] if [ "$servers" != "1" ]; then [2026-10-04T10:30:56.649Z] echo "Not all servers are running on the same node!" [2026-10-04T10:30:56.649Z] echo "$pcsstatus" [2026-10-04T10:30:56.649Z] ret=1 [2026-10-04T10:30:56.649Z] fi [2026-10-04T10:30:56.649Z] echo "$hypervisor" [2026-10-04T10:30:56.649Z] exit $ret [2026-10-04T10:30:56.649Z] ] *** [2026-10-04T10:30:56.649Z] ok: [an-a01n01] [2026-10-04T10:30:56.649Z] [2026-10-04T10:30:56.649Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] => (item=1) [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] => (item=2) [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] => (item=3) [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] => (item=4) [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] => (item=5) [2026-10-04T10:30:56.649Z] skipping: [an-a01n02] [2026-10-04T10:30:56.649Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-10-04T10:30:56.649Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-10-04T10:30:56.649Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-10-04T10:30:56.649Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-10-04T10:30:56.649Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-10-04T10:30:56.649Z] [2026-10-04T10:30:56.649Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:30:56.649Z] ] *** [2026-10-04T10:30:57.388Z] ok: [an-a01n01] [2026-10-04T10:30:57.388Z] [2026-10-04T10:30:57.388Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:30:57.388Z] ] *** [2026-10-04T10:30:58.582Z] ok: [an-a01n01] [2026-10-04T10:30:58.582Z] [2026-10-04T10:30:58.582Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:30:58.582Z] ] *** [2026-10-04T10:30:59.320Z] ok: [an-a01n01] [2026-10-04T10:30:59.320Z] [2026-10-04T10:30:59.320Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:30:59.320Z] ] *** [2026-10-04T10:31:00.514Z] ok: [an-a01n01] [2026-10-04T10:31:00.514Z] [2026-10-04T10:31:00.514Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:31:00.514Z] ] *** [2026-10-04T10:31:00.858Z] ok: [an-a01n01] [2026-10-04T10:31:00.858Z] [2026-10-04T10:31:00.858Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-10-04T10:31:01.202Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:31:01.202Z] [2026-10-04T10:31:01.202Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n02 -> localhost] [2026-10-04T10:34:25.391Z] ok: [an-a01n01 -> localhost] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:34:25.391Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:34:25.391Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:34:25.391Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:34:25.391Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:34:25.391Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:34:25.391Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:34:25.391Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:34:25.391Z] echo "pcs:" [2026-10-04T10:34:25.391Z] echo "$pcsstatus" [2026-10-04T10:34:25.391Z] exit 1 [2026-10-04T10:34:25.391Z] fi [2026-10-04T10:34:25.391Z] exit 0 [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:34:25.391Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:34:25.391Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:34:25.391Z] ret=0 [2026-10-04T10:34:25.391Z] if [ "$servers" != "1" ]; then [2026-10-04T10:34:25.391Z] echo "Not all servers are running on the same node!" [2026-10-04T10:34:25.391Z] echo "$pcsstatus" [2026-10-04T10:34:25.391Z] ret=1 [2026-10-04T10:34:25.391Z] fi [2026-10-04T10:34:25.391Z] echo "$hypervisor" [2026-10-04T10:34:25.391Z] exit $ret [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-10-04T10:34:25.391Z] ok: [an-a01n01] [2026-10-04T10:34:25.391Z] ok: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-10-04T10:34:25.391Z] ] ********** [2026-10-04T10:34:25.391Z] skipping: [an-a01n02] [2026-10-04T10:34:25.391Z] changed: [an-a01n01] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-10-04T10:34:25.391Z] skipping: [an-a01n01] [2026-10-04T10:34:25.391Z] skipping: [an-a01n02] [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-10-04T10:34:25.391Z] skipping: [an-a01n01] [2026-10-04T10:34:25.391Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-10-04T10:34:25.391Z] [2026-10-04T10:34:25.391Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-10-04T10:34:25.391Z] ] *** [2026-10-04T10:35:34.973Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-10-04T10:35:34.973Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-10-04T10:35:34.973Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-10-04T10:35:34.973Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-10-04T10:35:34.973Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-10-04T10:35:34.973Z] ok: [an-a01n02] [2026-10-04T10:35:34.973Z] [2026-10-04T10:35:34.973Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-10-04T10:35:34.973Z] skipping: [an-a01n01] [2026-10-04T10:35:34.973Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-10-04T10:35:34.973Z] [2026-10-04T10:35:34.973Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:35:34.973Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:35:34.973Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:35:34.973Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:35:34.973Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:35:34.973Z] echo "pcs:" [2026-10-04T10:35:34.973Z] echo "$pcsstatus" [2026-10-04T10:35:34.973Z] exit 1 [2026-10-04T10:35:34.973Z] fi [2026-10-04T10:35:34.973Z] exit 0 [2026-10-04T10:35:34.973Z] ] *** [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-10-04T10:36:44.551Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-10-04T10:36:44.551Z] ok: [an-a01n02] [2026-10-04T10:36:44.551Z] [2026-10-04T10:36:44.551Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:36:44.551Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:36:44.551Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:36:44.551Z] ret=0 [2026-10-04T10:36:44.551Z] if [ "$servers" != "1" ]; then [2026-10-04T10:36:44.551Z] echo "Not all servers are running on the same node!" [2026-10-04T10:36:44.551Z] echo "$pcsstatus" [2026-10-04T10:36:44.551Z] ret=1 [2026-10-04T10:36:44.551Z] fi [2026-10-04T10:36:44.551Z] echo "$hypervisor" [2026-10-04T10:36:44.551Z] exit $ret [2026-10-04T10:36:44.551Z] ] *** [2026-10-04T10:36:44.551Z] ok: [an-a01n02] [2026-10-04T10:36:44.551Z] [2026-10-04T10:36:44.551Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] => (item=1) [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] => (item=2) [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] => (item=3) [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] => (item=4) [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] => (item=5) [2026-10-04T10:36:44.551Z] skipping: [an-a01n01] [2026-10-04T10:36:44.551Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-10-04T10:36:44.551Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-10-04T10:36:44.551Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-10-04T10:36:44.551Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-10-04T10:36:44.551Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-10-04T10:36:44.551Z] [2026-10-04T10:36:44.551Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:36:44.551Z] ] *** [2026-10-04T10:36:57.903Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-10-04T10:36:57.903Z] ok: [an-a01n02] [2026-10-04T10:36:57.903Z] [2026-10-04T10:36:57.903Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:36:57.903Z] ] *** [2026-10-04T10:36:58.641Z] ok: [an-a01n02] [2026-10-04T10:36:58.641Z] [2026-10-04T10:36:58.641Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:36:58.641Z] ] *** [2026-10-04T10:36:59.381Z] ok: [an-a01n02] [2026-10-04T10:36:59.381Z] [2026-10-04T10:36:59.381Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:36:59.381Z] ] *** [2026-10-04T10:37:00.120Z] ok: [an-a01n02] [2026-10-04T10:37:00.120Z] [2026-10-04T10:37:00.120Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T10:37:00.120Z] ] *** [2026-10-04T10:37:00.466Z] ok: [an-a01n02] [2026-10-04T10:37:00.466Z] [2026-10-04T10:37:00.466Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-10-04T10:37:00.810Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-10-04T10:37:00.810Z] [2026-10-04T10:37:00.810Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01 -> localhost] [2026-10-04T10:40:25.009Z] ok: [an-a01n02 -> localhost] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:40:25.009Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:40:25.009Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:40:25.009Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] PLAY [basic server tests] ****************************************************** [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] [2026-10-04T10:40:25.009Z] ok: [an-a01n02] [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:25.009Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:40:25.009Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:40:25.009Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:40:25.009Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-10-04T10:40:25.009Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:25.009Z] [2026-10-04T10:40:25.009Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-10-04T10:40:25.009Z] if [ -z "$storage" ]; then [2026-10-04T10:40:25.009Z] echo "Unable to detect shared storage" [2026-10-04T10:40:25.009Z] exit 1 [2026-10-04T10:40:25.009Z] fi [2026-10-04T10:40:25.009Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:40:25.009Z] ] *** [2026-10-04T10:40:31.046Z] changed: [an-a01n01] [2026-10-04T10:40:31.046Z] changed: [an-a01n02] [2026-10-04T10:40:31.046Z] [2026-10-04T10:40:31.046Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-10-04T10:40:31.046Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:31.046Z] [2026-10-04T10:40:31.046Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-10-04T10:40:31.046Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-10-04T10:40:31.046Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-10-04T10:40:31.046Z] echo "Unable to detect all deploy iso" [2026-10-04T10:40:31.046Z] echo "Detected isos" [2026-10-04T10:40:31.046Z] echo "$isos" [2026-10-04T10:40:31.046Z] exit 1 [2026-10-04T10:40:31.046Z] fi [2026-10-04T10:40:31.046Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-10-04T10:40:31.046Z] if [ -z "$installiso" ]; then [2026-10-04T10:40:31.046Z] echo "Unable to detect install iso" [2026-10-04T10:40:31.046Z] echo "Detected isos" [2026-10-04T10:40:31.046Z] echo "$isos" [2026-10-04T10:40:31.046Z] exit 1 [2026-10-04T10:40:31.046Z] fi [2026-10-04T10:40:31.046Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:40:31.046Z] ] *** [2026-10-04T10:40:35.919Z] changed: [an-a01n01] [2026-10-04T10:40:35.919Z] changed: [an-a01n02] [2026-10-04T10:40:35.919Z] [2026-10-04T10:40:35.919Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-10-04T10:40:35.919Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:35.919Z] [2026-10-04T10:40:35.919Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:40:35.919Z] ] *** [2026-10-04T10:40:37.112Z] skipping: [an-a01n02] [2026-10-04T10:40:37.112Z] ok: [an-a01n01] [2026-10-04T10:40:37.112Z] [2026-10-04T10:40:37.112Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T10:40:37.112Z] --ci-test \ [2026-10-04T10:40:37.112Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T10:40:37.112Z] --os almalinux9 \ [2026-10-04T10:40:37.112Z] --cpu 2 --ram 4G \ [2026-10-04T10:40:37.112Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T10:40:37.112Z] --install-media {{ installisouuid }} \ [2026-10-04T10:40:37.112Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T10:40:37.112Z] ] *** [2026-10-04T10:40:37.112Z] skipping: [an-a01n01] [2026-10-04T10:40:37.112Z] skipping: [an-a01n02] [2026-10-04T10:40:37.112Z] [2026-10-04T10:40:37.112Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T10:40:37.112Z] skipping: [an-a01n01] [2026-10-04T10:40:37.112Z] skipping: [an-a01n02] [2026-10-04T10:40:37.112Z] [2026-10-04T10:40:37.112Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:40:37.456Z] skipping: [an-a01n01] [2026-10-04T10:40:37.456Z] skipping: [an-a01n02] [2026-10-04T10:40:37.456Z] [2026-10-04T10:40:37.456Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T10:40:37.456Z] skipping: [an-a01n01] [2026-10-04T10:40:37.456Z] skipping: [an-a01n02] [2026-10-04T10:40:37.456Z] [2026-10-04T10:40:37.456Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T10:40:37.456Z] skipping: [an-a01n01] [2026-10-04T10:40:37.456Z] skipping: [an-a01n02] [2026-10-04T10:40:37.456Z] [2026-10-04T10:40:37.456Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-10-04T10:40:37.456Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:37.456Z] [2026-10-04T10:40:37.456Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:40:37.456Z] ] *** [2026-10-04T10:40:39.177Z] skipping: [an-a01n01] [2026-10-04T10:40:39.177Z] ok: [an-a01n02] [2026-10-04T10:40:39.177Z] [2026-10-04T10:40:39.177Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T10:40:39.177Z] --ci-test \ [2026-10-04T10:40:39.177Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T10:40:39.177Z] --os almalinux9 \ [2026-10-04T10:40:39.177Z] --cpu 2 --ram 4G \ [2026-10-04T10:40:39.177Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T10:40:39.177Z] --install-media {{ installisouuid }} \ [2026-10-04T10:40:39.177Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T10:40:39.177Z] ] *** [2026-10-04T10:40:39.177Z] skipping: [an-a01n01] [2026-10-04T10:40:39.177Z] skipping: [an-a01n02] [2026-10-04T10:40:39.177Z] [2026-10-04T10:40:39.177Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T10:40:39.177Z] skipping: [an-a01n01] [2026-10-04T10:40:39.177Z] skipping: [an-a01n02] [2026-10-04T10:40:39.177Z] [2026-10-04T10:40:39.177Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:40:39.177Z] skipping: [an-a01n01] [2026-10-04T10:40:39.177Z] skipping: [an-a01n02] [2026-10-04T10:40:39.177Z] [2026-10-04T10:40:39.177Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T10:40:39.177Z] skipping: [an-a01n01] [2026-10-04T10:40:39.177Z] skipping: [an-a01n02] [2026-10-04T10:40:39.177Z] [2026-10-04T10:40:39.177Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T10:40:39.521Z] skipping: [an-a01n01] [2026-10-04T10:40:39.521Z] skipping: [an-a01n02] [2026-10-04T10:40:39.521Z] [2026-10-04T10:40:39.521Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-10-04T10:40:39.521Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:39.521Z] [2026-10-04T10:40:39.521Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:40:39.521Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:40:39.521Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:40:39.521Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:40:39.521Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:40:39.521Z] echo "pcs:" [2026-10-04T10:40:39.521Z] echo "$pcsstatus" [2026-10-04T10:40:39.521Z] exit 1 [2026-10-04T10:40:39.521Z] fi [2026-10-04T10:40:39.521Z] exit 0 [2026-10-04T10:40:39.521Z] ] *** [2026-10-04T10:40:41.853Z] ok: [an-a01n01] [2026-10-04T10:40:41.853Z] ok: [an-a01n02] [2026-10-04T10:40:41.853Z] [2026-10-04T10:40:41.853Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:40:41.853Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:40:41.853Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:40:41.853Z] ret=0 [2026-10-04T10:40:41.853Z] if [ "$servers" != "1" ]; then [2026-10-04T10:40:41.853Z] echo "Not all servers are running on the same node!" [2026-10-04T10:40:41.853Z] echo "$pcsstatus" [2026-10-04T10:40:41.853Z] ret=1 [2026-10-04T10:40:41.853Z] fi [2026-10-04T10:40:41.853Z] echo "$hypervisor" [2026-10-04T10:40:41.853Z] exit $ret [2026-10-04T10:40:41.853Z] ] *** [2026-10-04T10:40:43.573Z] ok: [an-a01n01] [2026-10-04T10:40:43.573Z] ok: [an-a01n02] [2026-10-04T10:40:43.573Z] [2026-10-04T10:40:43.573Z] PLAY [delete extra server tests] *********************************************** [2026-10-04T10:40:43.573Z] [2026-10-04T10:40:43.573Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:40:45.903Z] ok: [an-a01n01] [2026-10-04T10:40:45.903Z] ok: [an-a01n02] [2026-10-04T10:40:45.903Z] [2026-10-04T10:40:45.903Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T10:40:45.903Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:40:45.903Z] [2026-10-04T10:40:45.903Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:40:45.903Z] ] *** [2026-10-04T10:40:49.787Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:40:49.787Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:40:49.787Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:40:49.787Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:40:49.787Z] [2026-10-04T10:40:49.787Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-10-04T10:40:49.787Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T10:40:49.787Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T10:40:49.787Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T10:40:49.787Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T10:40:49.787Z] [2026-10-04T10:40:49.787Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:40:49.787Z] ] *** [2026-10-04T10:40:51.504Z] skipping: [an-a01n01] [2026-10-04T10:40:51.504Z] ok: [an-a01n02] [2026-10-04T10:40:51.504Z] [2026-10-04T10:40:51.504Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T10:40:51.504Z] --force \ [2026-10-04T10:40:51.504Z] --server an-test-deploy{{ servernum }} [2026-10-04T10:40:51.504Z] ] *** [2026-10-04T10:40:57.543Z] skipping: [an-a01n01] [2026-10-04T10:40:57.543Z] changed: [an-a01n02] [2026-10-04T10:40:57.543Z] [2026-10-04T10:40:57.543Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:40:57.543Z] ] *** [2026-10-04T10:43:06.248Z] skipping: [an-a01n01] [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (58 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (57 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (56 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (55 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (54 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (53 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (52 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (51 retries left). [2026-10-04T10:43:06.248Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (50 retries left). [2026-10-04T10:43:06.248Z] ok: [an-a01n02] [2026-10-04T10:43:06.248Z] [2026-10-04T10:43:06.248Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T10:43:06.248Z] ] *** [2026-10-04T10:43:06.248Z] skipping: [an-a01n01] [2026-10-04T10:43:06.248Z] ok: [an-a01n02] [2026-10-04T10:43:06.248Z] [2026-10-04T10:43:06.248Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T10:43:06.248Z] ] *** [2026-10-04T10:43:06.593Z] skipping: [an-a01n01] [2026-10-04T10:43:06.593Z] ok: [an-a01n02] [2026-10-04T10:43:06.593Z] [2026-10-04T10:43:06.593Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:43:06.593Z] ] *** [2026-10-04T10:43:08.311Z] skipping: [an-a01n01] [2026-10-04T10:43:08.311Z] ok: [an-a01n02] [2026-10-04T10:43:08.311Z] [2026-10-04T10:43:08.311Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T10:43:08.311Z] --force \ [2026-10-04T10:43:08.311Z] --server an-test-deploy{{ servernum }} [2026-10-04T10:43:08.311Z] ] *** [2026-10-04T10:43:15.728Z] skipping: [an-a01n01] [2026-10-04T10:43:15.728Z] changed: [an-a01n02] [2026-10-04T10:43:15.728Z] [2026-10-04T10:43:15.728Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:43:15.728Z] ] *** [2026-10-04T10:44:13.743Z] skipping: [an-a01n01] [2026-10-04T10:44:13.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-10-04T10:44:13.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-10-04T10:44:13.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-10-04T10:44:13.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-10-04T10:44:13.743Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-10-04T10:44:13.743Z] ok: [an-a01n02] [2026-10-04T10:44:13.743Z] [2026-10-04T10:44:13.743Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T10:44:13.743Z] ] *** [2026-10-04T10:44:14.480Z] skipping: [an-a01n01] [2026-10-04T10:44:14.480Z] ok: [an-a01n02] [2026-10-04T10:44:14.480Z] [2026-10-04T10:44:14.480Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T10:44:14.480Z] ] *** [2026-10-04T10:44:15.217Z] skipping: [an-a01n01] [2026-10-04T10:44:15.217Z] ok: [an-a01n02] [2026-10-04T10:44:15.217Z] [2026-10-04T10:44:15.217Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:44:15.217Z] ] *** [2026-10-04T10:44:16.934Z] skipping: [an-a01n01] [2026-10-04T10:44:16.934Z] ok: [an-a01n02] [2026-10-04T10:44:16.934Z] [2026-10-04T10:44:16.934Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T10:44:16.934Z] --force \ [2026-10-04T10:44:16.934Z] --server an-test-deploy{{ servernum }} [2026-10-04T10:44:16.934Z] ] *** [2026-10-04T10:44:23.710Z] skipping: [an-a01n01] [2026-10-04T10:44:23.710Z] changed: [an-a01n02] [2026-10-04T10:44:23.710Z] [2026-10-04T10:44:23.710Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:44:23.710Z] ] *** [2026-10-04T10:45:33.529Z] skipping: [an-a01n01] [2026-10-04T10:45:33.529Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-10-04T10:45:33.529Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-10-04T10:45:33.529Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-10-04T10:45:33.529Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-10-04T10:45:33.529Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-10-04T10:45:33.529Z] ok: [an-a01n02] [2026-10-04T10:45:33.529Z] [2026-10-04T10:45:33.529Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T10:45:33.529Z] ] *** [2026-10-04T10:45:33.529Z] skipping: [an-a01n01] [2026-10-04T10:45:33.529Z] ok: [an-a01n02] [2026-10-04T10:45:33.529Z] [2026-10-04T10:45:33.529Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T10:45:33.529Z] ] *** [2026-10-04T10:45:33.529Z] skipping: [an-a01n01] [2026-10-04T10:45:33.529Z] ok: [an-a01n02] [2026-10-04T10:45:33.529Z] [2026-10-04T10:45:33.529Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:45:33.529Z] ] *** [2026-10-04T10:45:33.529Z] skipping: [an-a01n01] [2026-10-04T10:45:33.529Z] ok: [an-a01n02] [2026-10-04T10:45:33.529Z] [2026-10-04T10:45:33.529Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T10:45:33.529Z] --force \ [2026-10-04T10:45:33.529Z] --server an-test-deploy{{ servernum }} [2026-10-04T10:45:33.529Z] ] *** [2026-10-04T10:45:33.529Z] skipping: [an-a01n01] [2026-10-04T10:45:33.529Z] changed: [an-a01n02] [2026-10-04T10:45:33.529Z] [2026-10-04T10:45:33.529Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:45:33.529Z] ] *** [2026-10-04T10:46:31.543Z] skipping: [an-a01n01] [2026-10-04T10:46:31.543Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-10-04T10:46:31.543Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-10-04T10:46:31.543Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-10-04T10:46:31.543Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-10-04T10:46:31.543Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-10-04T10:46:31.543Z] ok: [an-a01n02] [2026-10-04T10:46:31.543Z] [2026-10-04T10:46:31.543Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T10:46:31.543Z] ] *** [2026-10-04T10:46:31.543Z] skipping: [an-a01n01] [2026-10-04T10:46:31.543Z] ok: [an-a01n02] [2026-10-04T10:46:31.543Z] [2026-10-04T10:46:31.543Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T10:46:31.543Z] ] *** [2026-10-04T10:46:31.886Z] skipping: [an-a01n01] [2026-10-04T10:46:31.886Z] ok: [an-a01n02] [2026-10-04T10:46:31.886Z] [2026-10-04T10:46:31.886Z] PLAY [create server for DR testing] ******************************************** [2026-10-04T10:46:31.886Z] [2026-10-04T10:46:31.886Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:46:33.603Z] ok: [an-a01n02] [2026-10-04T10:46:33.603Z] ok: [an-a01n01] [2026-10-04T10:46:33.603Z] [2026-10-04T10:46:33.603Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T10:46:33.946Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:33.946Z] [2026-10-04T10:46:33.946Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:46:33.946Z] ] *** [2026-10-04T10:46:37.829Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:46:37.829Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:46:37.829Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:46:37.829Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:46:37.829Z] [2026-10-04T10:46:37.829Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-10-04T10:46:37.829Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:37.829Z] [2026-10-04T10:46:37.829Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-10-04T10:46:37.829Z] if [ -z "$storage" ]; then [2026-10-04T10:46:37.829Z] echo "Unable to detect shared storage" [2026-10-04T10:46:37.829Z] exit 1 [2026-10-04T10:46:37.829Z] fi [2026-10-04T10:46:37.829Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:46:37.829Z] ] *** [2026-10-04T10:46:42.706Z] changed: [an-a01n01] [2026-10-04T10:46:42.706Z] changed: [an-a01n02] [2026-10-04T10:46:42.706Z] [2026-10-04T10:46:42.706Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-10-04T10:46:42.706Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:42.706Z] [2026-10-04T10:46:42.706Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-10-04T10:46:42.706Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-10-04T10:46:42.706Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-10-04T10:46:42.706Z] echo "Unable to detect all deploy iso" [2026-10-04T10:46:42.706Z] echo "Detected isos" [2026-10-04T10:46:42.706Z] echo "$isos" [2026-10-04T10:46:42.706Z] exit 1 [2026-10-04T10:46:42.706Z] fi [2026-10-04T10:46:42.706Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-10-04T10:46:42.706Z] if [ -z "$installiso" ]; then [2026-10-04T10:46:42.706Z] echo "Unable to detect install iso" [2026-10-04T10:46:42.706Z] echo "Detected isos" [2026-10-04T10:46:42.706Z] echo "$isos" [2026-10-04T10:46:42.706Z] exit 1 [2026-10-04T10:46:42.706Z] fi [2026-10-04T10:46:42.706Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:46:42.706Z] ] *** [2026-10-04T10:46:47.602Z] changed: [an-a01n01] [2026-10-04T10:46:47.602Z] changed: [an-a01n02] [2026-10-04T10:46:47.602Z] [2026-10-04T10:46:47.602Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-10-04T10:46:47.602Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:47.602Z] [2026-10-04T10:46:47.602Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:46:47.602Z] ] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] ok: [an-a01n01] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T10:46:49.320Z] --ci-test \ [2026-10-04T10:46:49.320Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T10:46:49.320Z] --os almalinux9 \ [2026-10-04T10:46:49.320Z] --cpu 2 --ram 4G \ [2026-10-04T10:46:49.320Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T10:46:49.320Z] --install-media {{ installisouuid }} \ [2026-10-04T10:46:49.320Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T10:46:49.320Z] ] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n01] [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n01] [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n01] [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n01] [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T10:46:49.320Z] skipping: [an-a01n01] [2026-10-04T10:46:49.320Z] skipping: [an-a01n02] [2026-10-04T10:46:49.320Z] [2026-10-04T10:46:49.320Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-10-04T10:46:49.664Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:49.664Z] [2026-10-04T10:46:49.664Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:46:49.664Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:46:49.664Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:46:49.664Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:46:49.664Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:46:49.664Z] echo "pcs:" [2026-10-04T10:46:49.664Z] echo "$pcsstatus" [2026-10-04T10:46:49.664Z] exit 1 [2026-10-04T10:46:49.664Z] fi [2026-10-04T10:46:49.664Z] exit 0 [2026-10-04T10:46:49.664Z] ] *** [2026-10-04T10:46:51.381Z] ok: [an-a01n02] [2026-10-04T10:46:51.381Z] ok: [an-a01n01] [2026-10-04T10:46:51.381Z] [2026-10-04T10:46:51.381Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:46:51.381Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:46:51.381Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:46:51.381Z] ret=0 [2026-10-04T10:46:51.381Z] if [ "$servers" != "1" ]; then [2026-10-04T10:46:51.381Z] echo "Not all servers are running on the same node!" [2026-10-04T10:46:51.381Z] echo "$pcsstatus" [2026-10-04T10:46:51.381Z] ret=1 [2026-10-04T10:46:51.381Z] fi [2026-10-04T10:46:51.381Z] echo "$hypervisor" [2026-10-04T10:46:51.381Z] exit $ret [2026-10-04T10:46:51.381Z] ] *** [2026-10-04T10:46:53.711Z] ok: [an-a01n01] [2026-10-04T10:46:53.711Z] ok: [an-a01n02] [2026-10-04T10:46:53.711Z] [2026-10-04T10:46:53.711Z] PLAY [DR tests] **************************************************************** [2026-10-04T10:46:53.711Z] skipping: no hosts matched [2026-10-04T10:46:53.711Z] [2026-10-04T10:46:53.711Z] PLAY [upgrade server test] ***************************************************** [2026-10-04T10:46:53.711Z] [2026-10-04T10:46:53.711Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:46:55.435Z] ok: [an-a01n01] [2026-10-04T10:46:55.435Z] ok: [an-a01n02] [2026-10-04T10:46:55.435Z] [2026-10-04T10:46:55.435Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T10:46:55.435Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:55.435Z] [2026-10-04T10:46:55.435Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T10:46:55.435Z] ] *** [2026-10-04T10:46:59.320Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T10:46:59.320Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T10:46:59.320Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T10:46:59.320Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T10:46:59.320Z] [2026-10-04T10:46:59.320Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-10-04T10:46:59.320Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-10-04T10:46:59.320Z] [2026-10-04T10:46:59.320Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-10-04T10:46:59.320Z] if [ -z "$storage" ]; then [2026-10-04T10:46:59.320Z] echo "Unable to detect shared storage" [2026-10-04T10:46:59.320Z] exit 1 [2026-10-04T10:46:59.320Z] fi [2026-10-04T10:46:59.320Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:46:59.320Z] ] *** [2026-10-04T10:47:05.357Z] changed: [an-a01n02] [2026-10-04T10:47:05.357Z] changed: [an-a01n01] [2026-10-04T10:47:05.357Z] [2026-10-04T10:47:05.357Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-10-04T10:47:05.357Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-10-04T10:47:05.357Z] [2026-10-04T10:47:05.357Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-10-04T10:47:05.357Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-10-04T10:47:05.357Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-10-04T10:47:05.357Z] echo "Unable to detect all deploy iso" [2026-10-04T10:47:05.357Z] echo "Detected isos" [2026-10-04T10:47:05.357Z] echo "$isos" [2026-10-04T10:47:05.357Z] exit 1 [2026-10-04T10:47:05.357Z] fi [2026-10-04T10:47:05.357Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-10-04T10:47:05.357Z] if [ -z "$installiso" ]; then [2026-10-04T10:47:05.357Z] echo "Unable to detect install iso" [2026-10-04T10:47:05.357Z] echo "Detected isos" [2026-10-04T10:47:05.357Z] echo "$isos" [2026-10-04T10:47:05.357Z] exit 1 [2026-10-04T10:47:05.357Z] fi [2026-10-04T10:47:05.357Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-10-04T10:47:05.357Z] ] *** [2026-10-04T10:47:10.229Z] changed: [an-a01n01] [2026-10-04T10:47:10.229Z] changed: [an-a01n02] [2026-10-04T10:47:10.229Z] [2026-10-04T10:47:10.229Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-10-04T10:47:10.229Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-10-04T10:47:10.229Z] [2026-10-04T10:47:10.229Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T10:47:10.229Z] ] *** [2026-10-04T10:47:11.421Z] skipping: [an-a01n02] [2026-10-04T10:47:11.421Z] ok: [an-a01n01] [2026-10-04T10:47:11.421Z] [2026-10-04T10:47:11.421Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage f4552d7c-c6ba-4d42-abc9-cb2a8e2fc16f _raw_params=anvil-provision-server -vv --log-secure \ [2026-10-04T10:47:11.421Z] --ci-test \ [2026-10-04T10:47:11.421Z] --name an-test-deploy{{ servernum }} \ [2026-10-04T10:47:11.421Z] --os almalinux9 \ [2026-10-04T10:47:11.421Z] --cpu 2 --ram 4G \ [2026-10-04T10:47:11.421Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-10-04T10:47:11.421Z] --install-media {{ installisouuid }} \ [2026-10-04T10:47:11.421Z] --driver-disc deploy{{ servernum }}.iso [2026-10-04T10:47:11.421Z] ] *** [2026-10-04T10:47:11.421Z] skipping: [an-a01n01] [2026-10-04T10:47:11.421Z] skipping: [an-a01n02] [2026-10-04T10:47:11.421Z] [2026-10-04T10:47:11.421Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-10-04T10:47:11.421Z] skipping: [an-a01n01] [2026-10-04T10:47:11.421Z] skipping: [an-a01n02] [2026-10-04T10:47:11.421Z] [2026-10-04T10:47:11.421Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-10-04T10:47:11.421Z] skipping: [an-a01n01] [2026-10-04T10:47:11.421Z] skipping: [an-a01n02] [2026-10-04T10:47:11.421Z] [2026-10-04T10:47:11.421Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-10-04T10:47:11.765Z] skipping: [an-a01n01] [2026-10-04T10:47:11.765Z] skipping: [an-a01n02] [2026-10-04T10:47:11.765Z] [2026-10-04T10:47:11.765Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-10-04T10:47:11.765Z] skipping: [an-a01n01] [2026-10-04T10:47:11.765Z] skipping: [an-a01n02] [2026-10-04T10:47:11.765Z] [2026-10-04T10:47:11.765Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-10-04T10:47:11.765Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-10-04T10:47:11.765Z] [2026-10-04T10:47:11.765Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:47:11.765Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-10-04T10:47:11.765Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-10-04T10:47:11.765Z] if [ "$numservers" != "$startedservers" ]; then [2026-10-04T10:47:11.765Z] echo "Detected $numservers but only $startedservers in Started state" [2026-10-04T10:47:11.765Z] echo "pcs:" [2026-10-04T10:47:11.765Z] echo "$pcsstatus" [2026-10-04T10:47:11.765Z] exit 1 [2026-10-04T10:47:11.765Z] fi [2026-10-04T10:47:11.765Z] exit 0 [2026-10-04T10:47:11.765Z] ] *** [2026-10-04T10:47:14.093Z] ok: [an-a01n01] [2026-10-04T10:47:14.093Z] ok: [an-a01n02] [2026-10-04T10:47:14.093Z] [2026-10-04T10:47:14.093Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-10-04T10:47:14.093Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-10-04T10:47:14.093Z] servers="$(echo "$hypervisor" | wc -l)" [2026-10-04T10:47:14.093Z] ret=0 [2026-10-04T10:47:14.093Z] if [ "$servers" != "1" ]; then [2026-10-04T10:47:14.093Z] echo "Not all servers are running on the same node!" [2026-10-04T10:47:14.093Z] echo "$pcsstatus" [2026-10-04T10:47:14.093Z] ret=1 [2026-10-04T10:47:14.093Z] fi [2026-10-04T10:47:14.093Z] echo "$hypervisor" [2026-10-04T10:47:14.093Z] exit $ret [2026-10-04T10:47:14.093Z] ] *** [2026-10-04T10:47:15.810Z] ok: [an-a01n01] [2026-10-04T10:47:15.810Z] ok: [an-a01n02] [2026-10-04T10:47:15.810Z] [2026-10-04T10:47:15.810Z] PLAY [enable upgrade repo] ***************************************************** [2026-10-04T10:47:15.810Z] [2026-10-04T10:47:15.810Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:47:18.857Z] ok: [an-a01n01] [2026-10-04T10:47:18.857Z] ok: [an-a01n02] [2026-10-04T10:47:18.857Z] ok: [an-striker01] [2026-10-04T10:47:18.857Z] ok: [an-striker02] [2026-10-04T10:47:18.857Z] [2026-10-04T10:47:18.857Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-10-04T10:47:18.857Z] dnf config-manager --set-enabled $updaterepo [2026-10-04T10:47:18.857Z] ] *** [2026-10-04T10:47:29.854Z] changed: [an-a01n01] [2026-10-04T10:47:29.854Z] changed: [an-a01n02] [2026-10-04T10:47:29.854Z] changed: [an-striker01] [2026-10-04T10:47:29.854Z] changed: [an-striker02] [2026-10-04T10:47:29.854Z] [2026-10-04T10:47:29.854Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-10-04T10:47:30.198Z] ok: [an-striker01] [2026-10-04T10:47:30.198Z] ok: [an-striker02] [2026-10-04T10:47:30.198Z] ok: [an-a01n01] [2026-10-04T10:47:30.198Z] ok: [an-a01n02] [2026-10-04T10:47:30.198Z] [2026-10-04T10:47:30.198Z] PLAY [start upgrade process] *************************************************** [2026-10-04T10:47:30.198Z] [2026-10-04T10:47:30.198Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T10:47:32.526Z] ok: [an-striker01] [2026-10-04T10:47:32.526Z] [2026-10-04T10:47:32.526Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-10-04T10:47:32.526Z] ] *** [2026-10-04T11:10:49.252Z] changed: [an-striker01] [2026-10-04T11:10:49.252Z] [2026-10-04T11:10:49.252Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-10-04T11:11:12.552Z] changed: [an-striker01] [2026-10-04T11:11:12.552Z] [2026-10-04T11:11:12.552Z] PLAY [reconnect all nodes] ***************************************************** [2026-10-04T11:11:12.552Z] [2026-10-04T11:11:12.552Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:11:14.882Z] ok: [an-a01n02] [2026-10-04T11:11:14.882Z] ok: [an-striker02] [2026-10-04T11:11:14.882Z] ok: [an-a01n01] [2026-10-04T11:11:14.882Z] ok: [an-striker01] [2026-10-04T11:11:14.882Z] [2026-10-04T11:11:14.882Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-10-04T11:11:14.882Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02 [2026-10-04T11:11:14.882Z] [2026-10-04T11:11:14.882Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-10-04T11:14:39.066Z] ok: [an-striker01 -> localhost] [2026-10-04T11:14:39.066Z] ok: [an-striker02 -> localhost] [2026-10-04T11:14:39.066Z] ok: [an-a01n01 -> localhost] [2026-10-04T11:14:39.066Z] ok: [an-a01n02 -> localhost] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-10-04T11:14:39.066Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-10-04T11:14:39.066Z] exit 1 [2026-10-04T11:14:39.066Z] fi [2026-10-04T11:14:39.066Z] ] *** [2026-10-04T11:14:39.066Z] changed: [an-a01n01] [2026-10-04T11:14:39.066Z] changed: [an-a01n02] [2026-10-04T11:14:39.066Z] changed: [an-striker01] [2026-10-04T11:14:39.066Z] changed: [an-striker02] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] PLAY [check all nodes status after upgrade] ************************************ [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:14:39.066Z] ok: [an-striker01] [2026-10-04T11:14:39.066Z] ok: [an-striker02] [2026-10-04T11:14:39.066Z] ok: [an-a01n02] [2026-10-04T11:14:39.066Z] ok: [an-a01n01] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-10-04T11:14:39.066Z] echo "perl-Log-Journald update not found!" [2026-10-04T11:14:39.066Z] exit 1 [2026-10-04T11:14:39.066Z] fi [2026-10-04T11:14:39.066Z] ] *** [2026-10-04T11:14:39.066Z] changed: [an-a01n02] [2026-10-04T11:14:39.066Z] changed: [an-a01n01] [2026-10-04T11:14:39.066Z] changed: [an-striker01] [2026-10-04T11:14:39.066Z] changed: [an-striker02] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] PLAY [check nodes status after upgrade] **************************************** [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:14:39.066Z] ok: [an-a01n02] [2026-10-04T11:14:39.066Z] ok: [an-a01n01] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-10-04T11:14:39.066Z] echo "akmod-drbd update not found!" [2026-10-04T11:14:39.066Z] exit 1 [2026-10-04T11:14:39.066Z] fi [2026-10-04T11:14:39.066Z] ] *** [2026-10-04T11:14:39.066Z] changed: [an-a01n01] [2026-10-04T11:14:39.066Z] changed: [an-a01n02] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] PLAY [check hosted node status after upgrade] ********************************** [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:14:39.066Z] ok: [an-a01n01] [2026-10-04T11:14:39.066Z] ok: [an-a01n02] [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T11:14:39.066Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T11:14:39.066Z] [2026-10-04T11:14:39.066Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T11:14:39.066Z] ] *** [2026-10-04T11:14:40.261Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T11:14:40.261Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T11:14:40.261Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T11:14:40.261Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T11:14:40.261Z] [2026-10-04T11:14:40.261Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-10-04T11:14:40.261Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-10-04T11:14:40.261Z] [2026-10-04T11:14:40.261Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-10-04T11:14:40.261Z] ] *** [2026-10-04T11:14:41.454Z] ok: [an-a01n01] [2026-10-04T11:14:41.454Z] ok: [an-a01n02] [2026-10-04T11:14:41.454Z] [2026-10-04T11:14:41.454Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-10-04T11:14:41.797Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-10-04T11:14:41.797Z] [2026-10-04T11:14:41.797Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-10-04T11:14:41.797Z] ] *** [2026-10-04T11:14:42.141Z] ok: [an-a01n01] [2026-10-04T11:14:42.141Z] ok: [an-a01n02] [2026-10-04T11:14:42.141Z] [2026-10-04T11:14:42.141Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-10-04T11:14:42.486Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-10-04T11:14:42.486Z] [2026-10-04T11:14:42.486Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-10-04T11:14:42.486Z] ] *** [2026-10-04T11:14:43.225Z] ok: [an-a01n02] [2026-10-04T11:14:43.225Z] ok: [an-a01n01] [2026-10-04T11:14:43.225Z] [2026-10-04T11:14:43.225Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-10-04T11:14:43.225Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-10-04T11:14:43.225Z] [2026-10-04T11:14:43.225Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-10-04T11:14:43.225Z] ] *** [2026-10-04T11:14:43.963Z] ok: [an-a01n01] [2026-10-04T11:14:43.963Z] ok: [an-a01n02] [2026-10-04T11:14:43.963Z] [2026-10-04T11:14:43.963Z] PLAY [test anvil package self update] ****************************************** [2026-10-04T11:14:43.963Z] [2026-10-04T11:14:43.963Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:14:46.294Z] ok: [an-a01n01] [2026-10-04T11:14:46.294Z] ok: [an-a01n02] [2026-10-04T11:14:46.294Z] ok: [an-striker02] [2026-10-04T11:14:46.294Z] ok: [an-striker01] [2026-10-04T11:14:46.294Z] [2026-10-04T11:14:46.294Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-10-04T11:14:49.342Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-10-04T11:14:49.342Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-10-04T11:14:49.342Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-10-04T11:14:49.342Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-10-04T11:14:49.342Z] changed: [an-a01n02] => (item=scancore.service) [2026-10-04T11:14:49.342Z] changed: [an-a01n01] => (item=scancore.service) [2026-10-04T11:14:49.342Z] changed: [an-striker02] => (item=scancore.service) [2026-10-04T11:14:49.342Z] changed: [an-striker01] => (item=scancore.service) [2026-10-04T11:14:49.342Z] [2026-10-04T11:14:49.342Z] PLAY [test striker package self update] **************************************** [2026-10-04T11:14:49.342Z] [2026-10-04T11:14:49.342Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:14:50.535Z] ok: [an-striker02] [2026-10-04T11:14:50.535Z] ok: [an-striker01] [2026-10-04T11:14:50.535Z] [2026-10-04T11:14:50.535Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-10-04T11:14:50.535Z] ] *** [2026-10-04T11:15:18.559Z] changed: [an-striker01] [2026-10-04T11:15:18.559Z] changed: [an-striker02] [2026-10-04T11:15:18.559Z] [2026-10-04T11:15:18.559Z] PLAY [test node package self update] ******************************************* [2026-10-04T11:15:18.559Z] [2026-10-04T11:15:18.559Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:15:18.559Z] ok: [an-a01n01] [2026-10-04T11:15:18.559Z] ok: [an-a01n02] [2026-10-04T11:15:18.559Z] [2026-10-04T11:15:18.559Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-10-04T11:15:18.559Z] ] *** [2026-10-04T11:15:41.886Z] changed: [an-a01n02] [2026-10-04T11:15:41.886Z] changed: [an-a01n01] [2026-10-04T11:15:41.886Z] [2026-10-04T11:15:41.886Z] PLAY [test dr package self update] ********************************************* [2026-10-04T11:15:41.886Z] skipping: no hosts matched [2026-10-04T11:15:41.886Z] [2026-10-04T11:15:41.886Z] PLAY [test anvil package self update] ****************************************** [2026-10-04T11:15:41.886Z] [2026-10-04T11:15:41.886Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:15:41.886Z] ok: [an-a01n01] [2026-10-04T11:15:41.886Z] ok: [an-striker01] [2026-10-04T11:15:41.886Z] ok: [an-a01n02] [2026-10-04T11:15:41.886Z] ok: [an-striker02] [2026-10-04T11:15:41.886Z] [2026-10-04T11:15:41.886Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-10-04T11:15:44.216Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-10-04T11:15:44.216Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-10-04T11:15:44.216Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-10-04T11:15:44.216Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-10-04T11:15:44.216Z] changed: [an-a01n02] => (item=scancore.service) [2026-10-04T11:15:44.216Z] changed: [an-a01n01] => (item=scancore.service) [2026-10-04T11:15:44.216Z] changed: [an-striker01] => (item=scancore.service) [2026-10-04T11:15:44.216Z] changed: [an-striker02] => (item=scancore.service) [2026-10-04T11:15:44.216Z] [2026-10-04T11:15:44.216Z] PLAY [delete server tests] ***************************************************** [2026-10-04T11:15:44.216Z] [2026-10-04T11:15:44.216Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:15:45.933Z] ok: [an-a01n02] [2026-10-04T11:15:45.933Z] ok: [an-a01n01] [2026-10-04T11:15:45.933Z] [2026-10-04T11:15:45.933Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-10-04T11:15:45.933Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-10-04T11:15:45.933Z] [2026-10-04T11:15:45.933Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-10-04T11:15:45.933Z] ] *** [2026-10-04T11:15:49.818Z] ok: [an-a01n01] => (item=an-a01n01) [2026-10-04T11:15:49.818Z] ok: [an-a01n02] => (item=an-a01n01) [2026-10-04T11:15:49.818Z] ok: [an-a01n01] => (item=an-a01n02) [2026-10-04T11:15:49.818Z] ok: [an-a01n02] => (item=an-a01n02) [2026-10-04T11:15:49.818Z] [2026-10-04T11:15:49.818Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-10-04T11:15:50.162Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-10-04T11:15:50.162Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-10-04T11:15:50.162Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-10-04T11:15:50.162Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-10-04T11:15:50.162Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-10-04T11:15:50.162Z] [2026-10-04T11:15:50.162Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:15:50.162Z] ] *** [2026-10-04T11:15:51.881Z] skipping: [an-a01n01] [2026-10-04T11:15:51.881Z] ok: [an-a01n02] [2026-10-04T11:15:51.881Z] [2026-10-04T11:15:51.881Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T11:15:51.881Z] --force \ [2026-10-04T11:15:51.881Z] --server an-test-deploy{{ servernum }} [2026-10-04T11:15:51.881Z] ] *** [2026-10-04T11:15:57.920Z] skipping: [an-a01n01] [2026-10-04T11:15:57.920Z] changed: [an-a01n02] [2026-10-04T11:15:57.920Z] [2026-10-04T11:15:57.920Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:15:57.920Z] ] *** [2026-10-04T11:16:46.307Z] skipping: [an-a01n01] [2026-10-04T11:16:46.307Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-10-04T11:16:46.307Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-10-04T11:16:46.307Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-10-04T11:16:46.307Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-10-04T11:16:46.307Z] ok: [an-a01n02] [2026-10-04T11:16:46.307Z] [2026-10-04T11:16:46.307Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T11:16:46.307Z] ] *** [2026-10-04T11:16:46.307Z] skipping: [an-a01n01] [2026-10-04T11:16:46.307Z] ok: [an-a01n02] [2026-10-04T11:16:46.307Z] [2026-10-04T11:16:46.307Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T11:16:46.307Z] ] *** [2026-10-04T11:16:46.651Z] skipping: [an-a01n01] [2026-10-04T11:16:46.651Z] ok: [an-a01n02] [2026-10-04T11:16:46.651Z] [2026-10-04T11:16:46.651Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:46.651Z] ] *** [2026-10-04T11:16:48.372Z] skipping: [an-a01n01] [2026-10-04T11:16:48.372Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.919974", "end": "2026-10-04 07:16:48.000295", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 07:16:47.080321", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T11:16:48.372Z] ...ignoring [2026-10-04T11:16:48.372Z] [2026-10-04T11:16:48.372Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T11:16:48.372Z] --force \ [2026-10-04T11:16:48.372Z] --server an-test-deploy{{ servernum }} [2026-10-04T11:16:48.372Z] ] *** [2026-10-04T11:16:48.372Z] skipping: [an-a01n01] [2026-10-04T11:16:48.372Z] skipping: [an-a01n02] [2026-10-04T11:16:48.372Z] [2026-10-04T11:16:48.372Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:48.372Z] ] *** [2026-10-04T11:16:48.372Z] skipping: [an-a01n01] [2026-10-04T11:16:48.372Z] skipping: [an-a01n02] [2026-10-04T11:16:48.372Z] [2026-10-04T11:16:48.372Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T11:16:48.372Z] ] *** [2026-10-04T11:16:48.372Z] skipping: [an-a01n01] [2026-10-04T11:16:48.372Z] skipping: [an-a01n02] [2026-10-04T11:16:48.372Z] [2026-10-04T11:16:48.372Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T11:16:48.372Z] ] *** [2026-10-04T11:16:48.372Z] skipping: [an-a01n01] [2026-10-04T11:16:48.372Z] skipping: [an-a01n02] [2026-10-04T11:16:48.372Z] [2026-10-04T11:16:48.372Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:48.372Z] ] *** [2026-10-04T11:16:50.092Z] skipping: [an-a01n01] [2026-10-04T11:16:50.092Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.886646", "end": "2026-10-04 07:16:49.794407", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 07:16:48.907761", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T11:16:50.092Z] ...ignoring [2026-10-04T11:16:50.092Z] [2026-10-04T11:16:50.092Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T11:16:50.092Z] --force \ [2026-10-04T11:16:50.092Z] --server an-test-deploy{{ servernum }} [2026-10-04T11:16:50.092Z] ] *** [2026-10-04T11:16:50.092Z] skipping: [an-a01n01] [2026-10-04T11:16:50.092Z] skipping: [an-a01n02] [2026-10-04T11:16:50.092Z] [2026-10-04T11:16:50.092Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:50.092Z] ] *** [2026-10-04T11:16:50.092Z] skipping: [an-a01n01] [2026-10-04T11:16:50.092Z] skipping: [an-a01n02] [2026-10-04T11:16:50.092Z] [2026-10-04T11:16:50.092Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T11:16:50.092Z] ] *** [2026-10-04T11:16:50.092Z] skipping: [an-a01n01] [2026-10-04T11:16:50.092Z] skipping: [an-a01n02] [2026-10-04T11:16:50.092Z] [2026-10-04T11:16:50.092Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T11:16:50.092Z] ] *** [2026-10-04T11:16:50.436Z] skipping: [an-a01n01] [2026-10-04T11:16:50.436Z] skipping: [an-a01n02] [2026-10-04T11:16:50.436Z] [2026-10-04T11:16:50.436Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:50.436Z] ] *** [2026-10-04T11:16:52.159Z] skipping: [an-a01n01] [2026-10-04T11:16:52.159Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.981173", "end": "2026-10-04 07:16:51.660784", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 07:16:50.679611", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T11:16:52.159Z] ...ignoring [2026-10-04T11:16:52.159Z] [2026-10-04T11:16:52.159Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T11:16:52.159Z] --force \ [2026-10-04T11:16:52.159Z] --server an-test-deploy{{ servernum }} [2026-10-04T11:16:52.159Z] ] *** [2026-10-04T11:16:52.159Z] skipping: [an-a01n01] [2026-10-04T11:16:52.159Z] skipping: [an-a01n02] [2026-10-04T11:16:52.159Z] [2026-10-04T11:16:52.159Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:52.159Z] ] *** [2026-10-04T11:16:52.159Z] skipping: [an-a01n01] [2026-10-04T11:16:52.159Z] skipping: [an-a01n02] [2026-10-04T11:16:52.159Z] [2026-10-04T11:16:52.159Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T11:16:52.159Z] ] *** [2026-10-04T11:16:52.159Z] skipping: [an-a01n01] [2026-10-04T11:16:52.159Z] skipping: [an-a01n02] [2026-10-04T11:16:52.159Z] [2026-10-04T11:16:52.159Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T11:16:52.159Z] ] *** [2026-10-04T11:16:52.159Z] skipping: [an-a01n01] [2026-10-04T11:16:52.159Z] skipping: [an-a01n02] [2026-10-04T11:16:52.159Z] [2026-10-04T11:16:52.159Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:52.159Z] ] *** [2026-10-04T11:16:53.879Z] skipping: [an-a01n01] [2026-10-04T11:16:53.879Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.881039", "end": "2026-10-04 07:16:53.499191", "msg": "non-zero return code", "rc": 1, "start": "2026-10-04 07:16:52.618152", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-10-04T11:16:53.879Z] ...ignoring [2026-10-04T11:16:53.879Z] [2026-10-04T11:16:53.879Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-10-04T11:16:53.879Z] --force \ [2026-10-04T11:16:53.879Z] --server an-test-deploy{{ servernum }} [2026-10-04T11:16:53.879Z] ] *** [2026-10-04T11:16:53.879Z] skipping: [an-a01n01] [2026-10-04T11:16:53.879Z] skipping: [an-a01n02] [2026-10-04T11:16:53.879Z] [2026-10-04T11:16:53.879Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-10-04T11:16:53.879Z] ] *** [2026-10-04T11:16:53.879Z] skipping: [an-a01n01] [2026-10-04T11:16:53.879Z] skipping: [an-a01n02] [2026-10-04T11:16:53.879Z] [2026-10-04T11:16:53.879Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-10-04T11:16:53.879Z] ] *** [2026-10-04T11:16:53.879Z] skipping: [an-a01n01] [2026-10-04T11:16:53.879Z] skipping: [an-a01n02] [2026-10-04T11:16:53.879Z] [2026-10-04T11:16:53.879Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-10-04T11:16:53.879Z] ] *** [2026-10-04T11:16:54.224Z] skipping: [an-a01n01] [2026-10-04T11:16:54.224Z] skipping: [an-a01n02] [2026-10-04T11:16:54.224Z] [2026-10-04T11:16:54.224Z] PLAY [Check anvil available resources from striker01] ************************** [2026-10-04T11:16:54.224Z] [2026-10-04T11:16:54.224Z] TASK [Gathering Facts ] ******************************************************** [2026-10-04T11:16:56.559Z] ok: [an-striker01] [2026-10-04T11:16:56.559Z] [2026-10-04T11:16:56.559Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-10-04T11:16:56.559Z] sleep 300 [2026-10-04T11:16:56.559Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-10-04T11:16:56.559Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-10-04T11:16:56.559Z] ] *** [2026-10-04T11:22:06.391Z] ok: [an-striker01] [2026-10-04T11:22:06.391Z] [2026-10-04T11:22:06.391Z] PLAY RECAP ********************************************************************* [2026-10-04T11:22:06.391Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-10-04T11:22:06.391Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-10-04T11:22:06.391Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-10-04T11:22:06.391Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-10-04T11:22:06.391Z] [2026-10-04T11:22:06.391Z] tar: Removing leading `/' from member names [2026-10-04T11:22:06.391Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-10-04T11:22:06.391Z] /root/ci-tools [2026-10-04T11:22:06.391Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/1312.tar.gz [2026-10-04T11:22:06.391Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-10-04T11:22:06.391Z] This might take a few seconds. [2026-10-04T11:22:06.391Z] [2026-10-04T11:22:06.391Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-10-04T11:22:06.391Z] 𐄂 Disconnected from Red Hat Insights [2026-10-04T11:22:07.151Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-10-04T11:22:07.151Z] [2026-10-04T11:22:07.151Z] Manage your connected systems: https://red.ht/connector [2026-10-04T11:22:08.866Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-10-04T11:22:08.866Z] [2026-10-04T11:22:08.866Z] 1. Register with subscription-manager [2026-10-04T11:22:08.866Z] # subscription-manager register [2026-10-04T11:22:08.866Z] [2026-10-04T11:22:08.866Z] 2. Register with insights-client [2026-10-04T11:22:08.866Z] # insights-client --register [2026-10-04T11:22:10.057Z] This system is currently not registered. [2026-10-04T11:22:10.402Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-10-04T11:22:10.402Z] This might take a few seconds. [2026-10-04T11:22:10.402Z] [2026-10-04T11:22:10.402Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-10-04T11:22:12.732Z] 𐄂 Disconnected from Red Hat Insights [2026-10-04T11:22:13.925Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-10-04T11:22:13.925Z] [2026-10-04T11:22:13.925Z] Manage your connected systems: https://red.ht/connector [2026-10-04T11:22:16.255Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-10-04T11:22:16.255Z] [2026-10-04T11:22:16.255Z] 1. Register with subscription-manager [2026-10-04T11:22:16.255Z] # subscription-manager register [2026-10-04T11:22:16.255Z] [2026-10-04T11:22:16.255Z] 2. Register with insights-client [2026-10-04T11:22:16.255Z] # insights-client --register [2026-10-04T11:22:16.994Z] This system is currently not registered. [2026-10-04T11:22:17.337Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-10-04T11:22:17.337Z] This might take a few seconds. [2026-10-04T11:22:17.337Z] [2026-10-04T11:22:17.337Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-10-04T11:22:19.056Z] 𐄂 Disconnected from Red Hat Insights [2026-10-04T11:22:19.794Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-10-04T11:22:19.794Z] [2026-10-04T11:22:19.794Z] Manage your connected systems: https://red.ht/connector [2026-10-04T11:22:21.513Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-10-04T11:22:21.513Z] [2026-10-04T11:22:21.513Z] 1. Register with subscription-manager [2026-10-04T11:22:21.513Z] # subscription-manager register [2026-10-04T11:22:21.513Z] [2026-10-04T11:22:21.513Z] 2. Register with insights-client [2026-10-04T11:22:21.513Z] # insights-client --register [2026-10-04T11:22:22.250Z] This system is currently not registered. [2026-10-04T11:22:22.596Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-10-04T11:22:22.596Z] This might take a few seconds. [2026-10-04T11:22:22.596Z] [2026-10-04T11:22:22.596Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-10-04T11:22:24.926Z] 𐄂 Disconnected from Red Hat Insights [2026-10-04T11:22:25.270Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-10-04T11:22:25.270Z] [2026-10-04T11:22:25.270Z] Manage your connected systems: https://red.ht/connector [2026-10-04T11:22:26.989Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-10-04T11:22:26.989Z] [2026-10-04T11:22:26.989Z] 1. Register with subscription-manager [2026-10-04T11:22:26.989Z] # subscription-manager register [2026-10-04T11:22:26.989Z] [2026-10-04T11:22:26.989Z] 2. Register with insights-client [2026-10-04T11:22:26.989Z] # insights-client --register [2026-10-04T11:22:27.726Z] This system is currently not registered. [2026-10-04T11:22:28.464Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-10-04T11:22:28.464Z] This might take a few seconds. [2026-10-04T11:22:28.464Z] [2026-10-04T11:22:28.464Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-10-04T11:22:30.182Z] 𐄂 Disconnected from Red Hat Insights [2026-10-04T11:22:31.376Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-10-04T11:22:31.376Z] [2026-10-04T11:22:31.376Z] Manage your connected systems: https://red.ht/connector [2026-10-04T11:22:32.569Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-10-04T11:22:32.569Z] [2026-10-04T11:22:32.569Z] 1. Register with subscription-manager [2026-10-04T11:22:32.569Z] # subscription-manager register [2026-10-04T11:22:32.569Z] [2026-10-04T11:22:32.569Z] 2. Register with insights-client [2026-10-04T11:22:32.569Z] # insights-client --register [2026-10-04T11:22:33.763Z] This system is currently not registered. [2026-10-04T11:22:33.834Z] [Pipeline] } [2026-10-04T11:22:33.837Z] [Pipeline] // timeout [2026-10-04T11:22:33.841Z] [Pipeline] } [2026-10-04T11:22:33.843Z] [Pipeline] // script [2026-10-04T11:22:33.846Z] [Pipeline] } [2026-10-04T11:22:33.848Z] [Pipeline] // stage [2026-10-04T11:22:33.851Z] [Pipeline] stage [2026-10-04T11:22:33.851Z] [Pipeline] { (Declarative: Post Actions) [2026-10-04T11:22:33.858Z] [Pipeline] cleanWs [2026-10-04T11:22:33.907Z] [WS-CLEANUP] Deleting project workspace... [2026-10-04T11:22:33.907Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-10-04T11:22:33.932Z] [WS-CLEANUP] done [2026-10-04T11:22:33.936Z] [Pipeline] script [2026-10-04T11:22:33.937Z] [Pipeline] { [2026-10-04T11:22:33.954Z] [Pipeline] stage [2026-10-04T11:22:33.954Z] [Pipeline] { (Copying logs) [2026-10-04T11:22:33.998Z] [Pipeline] node [2026-10-04T11:22:34.137Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-10-04T11:22:34.138Z] [Pipeline] { [2026-10-04T11:22:34.195Z] [Pipeline] node [2026-10-04T11:22:34.331Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-10-04T11:22:34.332Z] [Pipeline] { [2026-10-04T11:22:34.336Z] [Pipeline] sh [2026-10-04T11:22:34.592Z] + mkdir -p /var/lib/jenkins/locks [2026-10-04T11:22:34.601Z] [Pipeline] echo [2026-10-04T11:22:34.601Z] RWLock: FD for lock log_archive in stage projectFinishUp is 644 [2026-10-04T11:22:34.605Z] [Pipeline] sleep [2026-10-04T11:22:34.605Z] No need to sleep any longer [2026-10-04T11:22:34.607Z] [Pipeline] echo [2026-10-04T11:22:34.608Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-10-04T11:22:34.610Z] [Pipeline] } [2026-10-04T11:22:34.613Z] [Pipeline] // node [2026-10-04T11:22:34.617Z] [Pipeline] echo [2026-10-04T11:22:34.617Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1312/ [2026-10-04T11:22:34.618Z] [Pipeline] sh [2026-10-04T11:22:34.874Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1312/ [2026-10-04T11:22:34.875Z] [Pipeline] sh