[2026-11-19T08:46:48.199Z] Started by upstream project "anvil/anvil-build-cron" build number 229 [2026-11-19T08:46:48.199Z] originally caused by: [2026-11-19T08:46:48.199Z] Started by timer [2026-11-19T08:46:49.401Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-11-19T08:46:49.412Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-11-19T08:46:49.701Z] Examining kronosnet/ci-tools [2026-11-19T08:46:49.701Z] Attempting to resolve main as a branch [2026-11-19T08:46:49.988Z] Resolved main as branch main at revision cb5604b20f65ce27f2f228b512e1c6096d73f983 [2026-11-19T08:46:49.989Z] Selected Git installation does not exist. Using Default [2026-11-19T08:46:49.989Z] The recommended git tool is: NONE [2026-11-19T08:46:49.989Z] using credential github-app-knet-jenkins [2026-11-19T08:46:49.992Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-11-19T08:46:49.992Z] Wiping out workspace first. [2026-11-19T08:46:50.020Z] Cloning the remote Git repository [2026-11-19T08:46:50.020Z] Using shallow clone with depth 1 [2026-11-19T08:46:50.020Z] Avoid fetching tags [2026-11-19T08:46:50.020Z] Honoring refspec on initial clone [2026-11-19T08:46:50.020Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-11-19T08:46:50.020Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@4 # timeout=10 [2026-11-19T08:46:50.030Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-11-19T08:46:50.030Z] > git --version # timeout=10 [2026-11-19T08:46:50.037Z] > git --version # 'git version 2.52.0' [2026-11-19T08:46:50.037Z] using GIT_ASKPASS to set credentials jenkins github integration (kronosnet org) [2026-11-19T08:46:50.043Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-11-19T08:46:50.298Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-11-19T08:46:50.308Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-11-19T08:46:50.317Z] Avoid second fetch [2026-11-19T08:46:50.317Z] Checking out Revision cb5604b20f65ce27f2f228b512e1c6096d73f983 (main) [2026-11-19T08:46:50.317Z] > git config core.sparsecheckout # timeout=10 [2026-11-19T08:46:50.324Z] > git checkout -f cb5604b20f65ce27f2f228b512e1c6096d73f983 # timeout=10 [2026-11-19T08:46:50.364Z] Commit message: "update github creds again" [2026-11-19T08:46:50.372Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-11-19T08:46:50.637Z] Examining kronosnet/ci-tools [2026-11-19T08:46:50.637Z] Attempting to resolve main as a branch [2026-11-19T08:46:50.823Z] Resolved main as branch main at revision cb5604b20f65ce27f2f228b512e1c6096d73f983 [2026-11-19T08:46:50.824Z] Selected Git installation does not exist. Using Default [2026-11-19T08:46:50.824Z] The recommended git tool is: NONE [2026-11-19T08:46:50.824Z] using credential github-app-knet-jenkins [2026-11-19T08:46:50.827Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-11-19T08:46:50.827Z] Wiping out workspace first. [2026-11-19T08:46:50.849Z] Cloning the remote Git repository [2026-11-19T08:46:50.849Z] Using shallow clone with depth 1 [2026-11-19T08:46:50.849Z] Avoid fetching tags [2026-11-19T08:46:50.849Z] Honoring refspec on initial clone [2026-11-19T08:46:50.849Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-11-19T08:46:50.849Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10 [2026-11-19T08:46:50.860Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-11-19T08:46:50.860Z] > git --version # timeout=10 [2026-11-19T08:46:50.868Z] > git --version # 'git version 2.52.0' [2026-11-19T08:46:50.868Z] using GIT_ASKPASS to set credentials jenkins github integration (kronosnet org) [2026-11-19T08:46:50.875Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-11-19T08:46:51.114Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-11-19T08:46:51.123Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-11-19T08:46:51.132Z] Avoid second fetch [2026-11-19T08:46:51.133Z] Checking out Revision cb5604b20f65ce27f2f228b512e1c6096d73f983 (main) [2026-11-19T08:46:51.133Z] > git config core.sparsecheckout # timeout=10 [2026-11-19T08:46:51.141Z] > git checkout -f cb5604b20f65ce27f2f228b512e1c6096d73f983 # timeout=10 [2026-11-19T08:46:51.182Z] Commit message: "update github creds again" [2026-11-19T08:46:51.247Z] [Pipeline] Start of Pipeline [2026-11-19T08:46:51.373Z] [Pipeline] node [2026-11-19T08:46:51.382Z] Running on anvil-ci-bm-phy03 in /srv/workspace/anvil/anvil-build-functional-testing [2026-11-19T08:46:51.383Z] [Pipeline] { [2026-11-19T08:46:51.393Z] [Pipeline] stage [2026-11-19T08:46:51.393Z] [Pipeline] { (Prep build env) [2026-11-19T08:46:51.403Z] [Pipeline] script [2026-11-19T08:46:51.405Z] [Pipeline] { [2026-11-19T08:46:51.409Z] [Pipeline] cleanWs [2026-11-19T08:46:51.458Z] [WS-CLEANUP] Deleting project workspace... [2026-11-19T08:46:51.458Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-11-19T08:46:51.482Z] [WS-CLEANUP] done [2026-11-19T08:46:51.505Z] [Pipeline] lock [2026-11-19T08:46:51.505Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-11-19T08:46:51.505Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-11-19T08:46:51.506Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT], resources: [MAINTENANCE LOCKOUT] [2026-11-19T08:46:51.506Z] [Pipeline] { [2026-11-19T08:46:51.509Z] [Pipeline] } [2026-11-19T08:46:51.509Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT], resources: [MAINTENANCE LOCKOUT] [2026-11-19T08:46:51.511Z] [Pipeline] // lock [2026-11-19T08:46:51.516Z] [Pipeline] } [2026-11-19T08:46:51.518Z] [Pipeline] // script [2026-11-19T08:46:51.521Z] [Pipeline] } [2026-11-19T08:46:51.523Z] [Pipeline] // stage [2026-11-19T08:46:51.526Z] [Pipeline] stage [2026-11-19T08:46:51.526Z] [Pipeline] { (Run functional testing) [2026-11-19T08:46:51.533Z] [Pipeline] script [2026-11-19T08:46:51.535Z] [Pipeline] { [2026-11-19T08:46:51.554Z] [Pipeline] echo [2026-11-19T08:46:51.554Z] runWithTimeout 360 [2026-11-19T08:46:51.555Z] [Pipeline] timeout [2026-11-19T08:46:51.555Z] Timeout set to expire in 6 hr 0 min [2026-11-19T08:46:51.555Z] [Pipeline] { [2026-11-19T08:46:51.560Z] [Pipeline] sh [2026-11-19T08:46:52.068Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm rhel-9 enterprise-staging none all [2026-11-19T08:46:52.068Z] CITBRANCH: main [2026-11-19T08:46:52.068Z] CITHOME: /root/ci-tools [2026-11-19T08:46:52.068Z] CITCMD: ci-setup-anvil-bm-vm [2026-11-19T08:46:52.068Z] CITCMDARGS: rhel-9 enterprise-staging none all [2026-11-19T08:46:52.068Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-11-19T08:46:52.408Z] Genering ssh keys and ssh config [2026-11-19T08:46:52.749Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-11-19T08:50:01.746Z] Destroying VMs (if any) [2026-11-19T08:50:01.746Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-11-19T08:50:01.746Z] Genering ssh keys and ssh config [2026-11-19T08:50:01.746Z] Cache already updated [2026-11-19T08:50:01.746Z] Creating nodes disks [2026-11-19T08:50:01.746Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/rhel-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-11-19T08:50:01.746Z] Generating an-test-deployX kickstart files/isos [2026-11-19T08:50:01.746Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-11-19T08:50:01.746Z] Media current: stdio file, overwriteable [2026-11-19T08:50:01.746Z] Media status : is blank [2026-11-19T08:50:01.746Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1714g free [2026-11-19T08:50:01.746Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] ISO image produced: 184 sectors [2026-11-19T08:50:01.746Z] Written to medium : 184 sectors at LBA 0 [2026-11-19T08:50:01.746Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-11-19T08:50:01.746Z] Media current: stdio file, overwriteable [2026-11-19T08:50:01.746Z] Media status : is blank [2026-11-19T08:50:01.746Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1722g free [2026-11-19T08:50:01.746Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] ISO image produced: 184 sectors [2026-11-19T08:50:01.746Z] Written to medium : 184 sectors at LBA 0 [2026-11-19T08:50:01.746Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-11-19T08:50:01.746Z] Media current: stdio file, overwriteable [2026-11-19T08:50:01.746Z] Media status : is blank [2026-11-19T08:50:01.746Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1722g free [2026-11-19T08:50:01.746Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] ISO image produced: 184 sectors [2026-11-19T08:50:01.746Z] Written to medium : 184 sectors at LBA 0 [2026-11-19T08:50:01.746Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-11-19T08:50:01.746Z] Media current: stdio file, overwriteable [2026-11-19T08:50:01.746Z] Media status : is blank [2026-11-19T08:50:01.746Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1722g free [2026-11-19T08:50:01.746Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] ISO image produced: 184 sectors [2026-11-19T08:50:01.746Z] Written to medium : 184 sectors at LBA 0 [2026-11-19T08:50:01.746Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-11-19T08:50:01.746Z] Media current: stdio file, overwriteable [2026-11-19T08:50:01.746Z] Media status : is blank [2026-11-19T08:50:01.746Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1722g free [2026-11-19T08:50:01.746Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-11-19T08:50:01.746Z] ISO image produced: 184 sectors [2026-11-19T08:50:01.746Z] Written to medium : 184 sectors at LBA 0 [2026-11-19T08:50:01.746Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-11-19T08:50:01.746Z] [2026-11-19T08:50:01.746Z] Generating cloud-init configs [2026-11-19T08:50:01.746Z] Generating cloud-init metadata and seed images [2026-11-19T08:50:01.746Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-11-19T08:50:01.746Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-11-19T08:50:01.746Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-11-19T08:50:01.746Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-11-19T08:50:02.087Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-11-19T08:50:02.087Z] Creating VMs [2026-11-19T08:50:03.272Z] [2026-11-19T08:50:03.272Z] Starting install... [2026-11-19T08:50:03.272Z] [2026-11-19T08:50:03.272Z] Creating domain... | 00:00 [2026-11-19T08:50:03.272Z] Domain creation completed. [2026-11-19T08:50:04.455Z] [2026-11-19T08:50:04.455Z] Starting install... [2026-11-19T08:50:04.455Z] [2026-11-19T08:50:04.455Z] Creating domain... | 00:00 [2026-11-19T08:50:04.455Z] Domain creation completed. [2026-11-19T08:50:05.640Z] [2026-11-19T08:50:05.640Z] Starting install... [2026-11-19T08:50:05.640Z] [2026-11-19T08:50:05.640Z] Creating domain... | 00:00 [2026-11-19T08:50:05.640Z] Domain creation completed. [2026-11-19T08:50:07.348Z] [2026-11-19T08:50:07.348Z] Starting install... [2026-11-19T08:50:07.348Z] [2026-11-19T08:50:07.348Z] Creating domain... | 00:00 [2026-11-19T08:50:07.348Z] Domain creation completed. [2026-11-19T08:50:08.534Z] [2026-11-19T08:50:08.534Z] Starting install... [2026-11-19T08:50:08.534Z] [2026-11-19T08:50:08.534Z] Creating domain... | 00:00 [2026-11-19T08:50:08.534Z] Domain creation completed. [2026-11-19T08:50:08.534Z] Waiting for nodes and cloud-init to complete full init [2026-11-19T08:50:12.403Z] cloud-init status node an-striker01: [2026-11-19T08:50:15.437Z] cloud-init status node an-striker02: [2026-11-19T08:50:18.470Z] cloud-init status node an-a01n01: [2026-11-19T08:50:21.503Z] cloud-init status node an-a01n02: [2026-11-19T08:50:24.541Z] cloud-init status node an-a01dr01: [2026-11-19T08:50:24.541Z] Thu Nov 19 03:50:24 EST 2026 actual: 0 expected: 5 [2026-11-19T08:50:35.572Z] cloud-init status node an-striker01: running [2026-11-19T08:50:36.346Z] cloud-init status node an-striker02: running [2026-11-19T08:50:40.642Z] cloud-init status node an-a01n01: running [2026-11-19T08:50:42.001Z] cloud-init status node an-a01n02: running [2026-11-19T08:50:42.786Z] cloud-init status node an-a01dr01: running [2026-11-19T08:50:42.786Z] Thu Nov 19 03:50:42 EST 2026 actual: 0 expected: 5 [2026-11-19T08:50:55.247Z] cloud-init status node an-striker01: running [2026-11-19T08:50:55.247Z] cloud-init status node an-striker02: running [2026-11-19T08:50:55.247Z] cloud-init status node an-a01n01: running [2026-11-19T08:50:56.207Z] cloud-init status node an-a01n02: running [2026-11-19T08:50:56.992Z] cloud-init status node an-a01dr01: running [2026-11-19T08:50:56.992Z] Thu Nov 19 03:50:56 EST 2026 actual: 0 expected: 5 [2026-11-19T08:51:09.205Z] cloud-init status node an-striker01: running [2026-11-19T08:51:09.205Z] cloud-init status node an-striker02: running [2026-11-19T08:51:09.602Z] cloud-init status node an-a01n01: running [2026-11-19T08:51:10.389Z] cloud-init status node an-a01n02: running [2026-11-19T08:51:11.668Z] cloud-init status node an-a01dr01: running [2026-11-19T08:51:11.668Z] Thu Nov 19 03:51:11 EST 2026 actual: 0 expected: 5 [2026-11-19T08:51:22.446Z] cloud-init status node an-striker01: running [2026-11-19T08:51:23.398Z] cloud-init status node an-striker02: running [2026-11-19T08:51:23.903Z] cloud-init status node an-a01n01: running [2026-11-19T08:51:24.788Z] cloud-init status node an-a01n02: running [2026-11-19T08:51:25.597Z] cloud-init status node an-a01dr01: running [2026-11-19T08:51:25.597Z] Thu Nov 19 03:51:25 EST 2026 actual: 0 expected: 5 [2026-11-19T08:51:37.528Z] cloud-init status node an-striker01: running [2026-11-19T08:51:37.884Z] cloud-init status node an-striker02: running [2026-11-19T08:51:39.121Z] cloud-init status node an-a01n01: running [2026-11-19T08:51:39.929Z] cloud-init status node an-a01n02: running [2026-11-19T08:51:40.863Z] cloud-init status node an-a01dr01: running [2026-11-19T08:51:40.863Z] Thu Nov 19 03:51:40 EST 2026 actual: 0 expected: 5 [2026-11-19T08:51:53.089Z] cloud-init status node an-striker01: running [2026-11-19T08:51:53.089Z] cloud-init status node an-striker02: running [2026-11-19T08:51:53.472Z] cloud-init status node an-a01n01: running [2026-11-19T08:51:54.677Z] cloud-init status node an-a01n02: running [2026-11-19T08:51:55.122Z] cloud-init status node an-a01dr01: running [2026-11-19T08:51:55.122Z] Thu Nov 19 03:51:55 EST 2026 actual: 0 expected: 5 [2026-11-19T08:52:07.723Z] cloud-init status node an-striker01: running [2026-11-19T08:52:07.723Z] cloud-init status node an-striker02: running [2026-11-19T08:52:09.372Z] cloud-init status node an-a01n01: running [2026-11-19T08:52:09.372Z] cloud-init status node an-a01n02: running [2026-11-19T08:52:10.041Z] cloud-init status node an-a01dr01: running [2026-11-19T08:52:10.041Z] Thu Nov 19 03:52:09 EST 2026 actual: 0 expected: 5 [2026-11-19T08:52:21.966Z] cloud-init status node an-striker01: running [2026-11-19T08:52:21.966Z] cloud-init status node an-striker02: running [2026-11-19T08:52:22.907Z] cloud-init status node an-a01n01: running [2026-11-19T08:52:24.155Z] cloud-init status node an-a01n02: running [2026-11-19T08:52:24.791Z] cloud-init status node an-a01dr01: running [2026-11-19T08:52:24.791Z] Thu Nov 19 03:52:24 EST 2026 actual: 0 expected: 5 [2026-11-19T08:52:36.229Z] cloud-init status node an-striker01: running [2026-11-19T08:52:36.229Z] cloud-init status node an-striker02: running [2026-11-19T08:52:37.111Z] cloud-init status node an-a01n01: running [2026-11-19T08:52:38.344Z] cloud-init status node an-a01n02: running [2026-11-19T08:52:39.187Z] cloud-init status node an-a01dr01: running [2026-11-19T08:52:39.187Z] Thu Nov 19 03:52:38 EST 2026 actual: 0 expected: 5 [2026-11-19T08:52:55.843Z] cloud-init status node an-striker01: running [2026-11-19T08:52:55.843Z] cloud-init status node an-striker02: running [2026-11-19T08:52:57.647Z] cloud-init status node an-a01n01: running [2026-11-19T08:52:57.647Z] cloud-init status node an-a01n02: running [2026-11-19T08:52:59.794Z] cloud-init status node an-a01dr01: running [2026-11-19T08:52:59.794Z] Thu Nov 19 03:52:58 EST 2026 actual: 0 expected: 5 [2026-11-19T08:53:09.914Z] cloud-init status node an-striker01: running [2026-11-19T08:53:10.429Z] cloud-init status node an-striker02: running [2026-11-19T08:53:11.382Z] cloud-init status node an-a01n01: running [2026-11-19T08:53:12.428Z] cloud-init status node an-a01n02: running [2026-11-19T08:53:12.802Z] cloud-init status node an-a01dr01: running [2026-11-19T08:53:12.802Z] Thu Nov 19 03:53:12 EST 2026 actual: 0 expected: 5 [2026-11-19T08:53:36.044Z] cloud-init status node an-striker01: running [2026-11-19T08:53:36.044Z] cloud-init status node an-striker02: running [2026-11-19T08:53:36.044Z] cloud-init status node an-a01n01: running [2026-11-19T08:53:36.402Z] cloud-init status node an-a01n02: running [2026-11-19T08:53:37.578Z] cloud-init status node an-a01dr01: running [2026-11-19T08:53:37.578Z] Thu Nov 19 03:53:37 EST 2026 actual: 0 expected: 5 [2026-11-19T08:53:49.324Z] cloud-init status node an-striker01: running [2026-11-19T08:53:49.324Z] cloud-init status node an-striker02: running [2026-11-19T08:53:50.504Z] cloud-init status node an-a01n01: running [2026-11-19T08:53:51.045Z] cloud-init status node an-a01n02: running [2026-11-19T08:53:52.259Z] cloud-init status node an-a01dr01: running [2026-11-19T08:53:52.259Z] Thu Nov 19 03:53:51 EST 2026 actual: 0 expected: 5 [2026-11-19T08:54:15.319Z] cloud-init status node an-striker01: [2026-11-19T08:54:15.319Z] cloud-init status node an-striker02: running [2026-11-19T08:54:15.319Z] cloud-init status node an-a01n01: running [2026-11-19T08:54:15.319Z] cloud-init status node an-a01n02: running [2026-11-19T08:54:15.855Z] cloud-init status node an-a01dr01: running [2026-11-19T08:54:15.855Z] Thu Nov 19 03:54:15 EST 2026 actual: 0 expected: 5 [2026-11-19T08:54:34.519Z] cloud-init status node an-striker01: running [2026-11-19T08:54:38.330Z] cloud-init status node an-striker02: running [2026-11-19T08:54:38.330Z] cloud-init status node an-a01n01: running [2026-11-19T08:54:39.288Z] cloud-init status node an-a01n02: running [2026-11-19T08:54:40.070Z] cloud-init status node an-a01dr01: running [2026-11-19T08:54:40.070Z] Thu Nov 19 03:54:39 EST 2026 actual: 0 expected: 5 [2026-11-19T08:54:51.223Z] cloud-init status node an-striker01: running [2026-11-19T08:54:52.468Z] cloud-init status node an-striker02: running [2026-11-19T08:54:58.595Z] cloud-init status node an-a01n01: running [2026-11-19T08:54:59.061Z] cloud-init status node an-a01n02: running [2026-11-19T08:54:59.635Z] cloud-init status node an-a01dr01: running [2026-11-19T08:54:59.635Z] Thu Nov 19 03:54:59 EST 2026 actual: 0 expected: 5 [2026-11-19T08:55:12.121Z] cloud-init status node an-striker01: running [2026-11-19T08:55:12.121Z] cloud-init status node an-striker02: running [2026-11-19T08:55:12.578Z] cloud-init status node an-a01n01: running [2026-11-19T08:55:18.405Z] cloud-init status node an-a01n02: running [2026-11-19T08:55:19.181Z] cloud-init status node an-a01dr01: running [2026-11-19T08:55:19.181Z] Thu Nov 19 03:55:19 EST 2026 actual: 0 expected: 5 [2026-11-19T08:55:30.934Z] cloud-init status node an-striker01: running [2026-11-19T08:55:36.603Z] cloud-init status node an-striker02: running [2026-11-19T08:55:36.959Z] cloud-init status node an-a01n01: running [2026-11-19T08:55:37.999Z] cloud-init status node an-a01n02: running [2026-11-19T08:55:38.831Z] cloud-init status node an-a01dr01: running [2026-11-19T08:55:38.831Z] Thu Nov 19 03:55:38 EST 2026 actual: 0 expected: 5 [2026-11-19T08:55:50.354Z] cloud-init status node an-striker01: running [2026-11-19T08:55:50.830Z] cloud-init status node an-striker02: running [2026-11-19T08:55:51.628Z] cloud-init status node an-a01n01: running [2026-11-19T08:55:52.407Z] cloud-init status node an-a01n02: running [2026-11-19T08:55:53.187Z] cloud-init status node an-a01dr01: running [2026-11-19T08:55:53.187Z] Thu Nov 19 03:55:53 EST 2026 actual: 0 expected: 5 [2026-11-19T08:56:04.800Z] cloud-init status node an-striker01: running [2026-11-19T08:56:05.157Z] cloud-init status node an-striker02: running [2026-11-19T08:56:06.361Z] cloud-init status node an-a01n01: running [2026-11-19T08:56:07.097Z] cloud-init status node an-a01n02: running [2026-11-19T08:56:07.832Z] cloud-init status node an-a01dr01: running [2026-11-19T08:56:07.832Z] Thu Nov 19 03:56:07 EST 2026 actual: 0 expected: 5 [2026-11-19T08:56:18.840Z] cloud-init status node an-striker01: running [2026-11-19T08:56:19.573Z] cloud-init status node an-striker02: running [2026-11-19T08:56:20.762Z] cloud-init status node an-a01n01: running [2026-11-19T08:56:21.495Z] cloud-init status node an-a01n02: running [2026-11-19T08:56:22.227Z] cloud-init status node an-a01dr01: running [2026-11-19T08:56:22.227Z] Thu Nov 19 03:56:22 EST 2026 actual: 0 expected: 5 [2026-11-19T08:56:33.196Z] cloud-init status node an-striker01: running [2026-11-19T08:56:34.382Z] cloud-init status node an-striker02: running [2026-11-19T08:56:35.116Z] cloud-init status node an-a01n01: running [2026-11-19T08:56:35.850Z] cloud-init status node an-a01n02: running [2026-11-19T08:56:37.038Z] cloud-init status node an-a01dr01: running [2026-11-19T08:56:37.038Z] Thu Nov 19 03:56:36 EST 2026 actual: 0 expected: 5 [2026-11-19T08:56:48.016Z] cloud-init status node an-striker01: running [2026-11-19T08:56:48.751Z] cloud-init status node an-striker02: running [2026-11-19T08:56:49.486Z] cloud-init status node an-a01n01: running [2026-11-19T08:56:50.673Z] cloud-init status node an-a01n02: running [2026-11-19T08:56:51.408Z] cloud-init status node an-a01dr01: running [2026-11-19T08:56:51.408Z] Thu Nov 19 03:56:51 EST 2026 actual: 0 expected: 5 [2026-11-19T08:57:02.411Z] cloud-init status node an-striker01: running [2026-11-19T08:57:03.600Z] cloud-init status node an-striker02: running [2026-11-19T08:57:04.337Z] cloud-init status node an-a01n01: done [2026-11-19T08:57:05.072Z] cloud-init status node an-a01n02: running [2026-11-19T08:57:06.262Z] cloud-init status node an-a01dr01: running [2026-11-19T08:57:06.262Z] Thu Nov 19 03:57:06 EST 2026 actual: 1 expected: 5 [2026-11-19T08:57:17.246Z] cloud-init status node an-striker01: running [2026-11-19T08:57:17.981Z] cloud-init status node an-striker02: running [2026-11-19T08:57:19.167Z] cloud-init status node an-a01n01: done [2026-11-19T08:57:19.902Z] cloud-init status node an-a01n02: running [2026-11-19T08:57:21.089Z] cloud-init status node an-a01dr01: running [2026-11-19T08:57:21.089Z] Thu Nov 19 03:57:20 EST 2026 actual: 1 expected: 5 [2026-11-19T08:57:32.065Z] cloud-init status node an-striker01: running [2026-11-19T08:57:32.800Z] cloud-init status node an-striker02: running [2026-11-19T08:57:33.987Z] cloud-init status node an-a01n01: done [2026-11-19T08:57:35.699Z] cloud-init status node an-a01n02: running [2026-11-19T08:57:36.434Z] cloud-init status node an-a01dr01: done [2026-11-19T08:57:36.434Z] Thu Nov 19 03:57:36 EST 2026 actual: 2 expected: 5 [2026-11-19T08:57:47.411Z] cloud-init status node an-striker01: running [2026-11-19T08:57:48.601Z] cloud-init status node an-striker02: running [2026-11-19T08:57:49.335Z] cloud-init status node an-a01n01: done [2026-11-19T08:57:50.522Z] cloud-init status node an-a01n02: running [2026-11-19T08:57:51.256Z] cloud-init status node an-a01dr01: done [2026-11-19T08:57:51.256Z] Thu Nov 19 03:57:50 EST 2026 actual: 2 expected: 5 [2026-11-19T08:58:02.230Z] cloud-init status node an-striker01: running [2026-11-19T08:58:02.964Z] cloud-init status node an-striker02: running [2026-11-19T08:58:04.895Z] cloud-init status node an-a01n01: done [2026-11-19T08:58:04.895Z] cloud-init status node an-a01n02: running [2026-11-19T08:58:05.978Z] cloud-init status node an-a01dr01: done [2026-11-19T08:58:05.978Z] Thu Nov 19 03:58:05 EST 2026 actual: 2 expected: 5 [2026-11-19T08:58:16.952Z] cloud-init status node an-striker01: running [2026-11-19T08:58:17.686Z] cloud-init status node an-striker02: running [2026-11-19T08:58:18.420Z] cloud-init status node an-a01n01: done [2026-11-19T08:58:19.154Z] cloud-init status node an-a01n02: running [2026-11-19T08:58:20.345Z] cloud-init status node an-a01dr01: done [2026-11-19T08:58:20.345Z] Thu Nov 19 03:58:19 EST 2026 actual: 2 expected: 5 [2026-11-19T08:58:31.389Z] cloud-init status node an-striker01: running [2026-11-19T08:58:32.576Z] cloud-init status node an-striker02: running [2026-11-19T08:58:33.767Z] cloud-init status node an-a01n01: done [2026-11-19T08:58:35.481Z] cloud-init status node an-a01n02: done [2026-11-19T08:58:36.675Z] cloud-init status node an-a01dr01: done [2026-11-19T08:58:36.675Z] Thu Nov 19 03:58:36 EST 2026 actual: 3 expected: 5 [2026-11-19T08:58:47.652Z] cloud-init status node an-striker01: running [2026-11-19T08:58:48.838Z] cloud-init status node an-striker02: running [2026-11-19T08:58:49.572Z] cloud-init status node an-a01n01: done [2026-11-19T08:58:50.759Z] cloud-init status node an-a01n02: done [2026-11-19T08:58:51.945Z] cloud-init status node an-a01dr01: done [2026-11-19T08:58:51.945Z] Thu Nov 19 03:58:51 EST 2026 actual: 3 expected: 5 [2026-11-19T08:59:05.234Z] cloud-init status node an-striker01: running [2026-11-19T08:59:05.234Z] cloud-init status node an-striker02: running [2026-11-19T08:59:05.576Z] cloud-init status node an-a01n01: done [2026-11-19T08:59:06.312Z] cloud-init status node an-a01n02: done [2026-11-19T08:59:07.500Z] cloud-init status node an-a01dr01: done [2026-11-19T08:59:07.500Z] Thu Nov 19 03:59:07 EST 2026 actual: 3 expected: 5 [2026-11-19T08:59:18.471Z] cloud-init status node an-striker01: running [2026-11-19T08:59:19.657Z] cloud-init status node an-striker02: running [2026-11-19T08:59:20.390Z] cloud-init status node an-a01n01: done [2026-11-19T08:59:21.124Z] cloud-init status node an-a01n02: done [2026-11-19T08:59:22.309Z] cloud-init status node an-a01dr01: done [2026-11-19T08:59:22.309Z] Thu Nov 19 03:59:21 EST 2026 actual: 3 expected: 5 [2026-11-19T08:59:33.279Z] cloud-init status node an-striker01: running [2026-11-19T08:59:34.011Z] cloud-init status node an-striker02: running [2026-11-19T08:59:34.744Z] cloud-init status node an-a01n01: done [2026-11-19T08:59:35.477Z] cloud-init status node an-a01n02: done [2026-11-19T08:59:36.660Z] cloud-init status node an-a01dr01: done [2026-11-19T08:59:36.660Z] Thu Nov 19 03:59:36 EST 2026 actual: 3 expected: 5 [2026-11-19T08:59:47.632Z] cloud-init status node an-striker01: running [2026-11-19T08:59:48.363Z] cloud-init status node an-striker02: running [2026-11-19T08:59:49.096Z] cloud-init status node an-a01n01: done [2026-11-19T08:59:50.284Z] cloud-init status node an-a01n02: done [2026-11-19T08:59:51.017Z] cloud-init status node an-a01dr01: done [2026-11-19T08:59:51.017Z] Thu Nov 19 03:59:50 EST 2026 actual: 3 expected: 5 [2026-11-19T09:00:01.988Z] cloud-init status node an-striker01: running [2026-11-19T09:00:02.722Z] cloud-init status node an-striker02: running [2026-11-19T09:00:03.906Z] cloud-init status node an-a01n01: done [2026-11-19T09:00:04.638Z] cloud-init status node an-a01n02: done [2026-11-19T09:00:05.825Z] cloud-init status node an-a01dr01: done [2026-11-19T09:00:05.825Z] Thu Nov 19 04:00:05 EST 2026 actual: 3 expected: 5 [2026-11-19T09:00:16.938Z] cloud-init status node an-striker01: running [2026-11-19T09:00:18.124Z] cloud-init status node an-striker02: running [2026-11-19T09:00:18.861Z] cloud-init status node an-a01n01: done [2026-11-19T09:00:19.593Z] cloud-init status node an-a01n02: done [2026-11-19T09:00:20.779Z] cloud-init status node an-a01dr01: done [2026-11-19T09:00:20.779Z] Thu Nov 19 04:00:20 EST 2026 actual: 3 expected: 5 [2026-11-19T09:00:31.750Z] cloud-init status node an-striker01: done [2026-11-19T09:00:32.938Z] cloud-init status node an-striker02: running [2026-11-19T09:00:33.671Z] cloud-init status node an-a01n01: done [2026-11-19T09:00:34.405Z] cloud-init status node an-a01n02: done [2026-11-19T09:00:35.138Z] cloud-init status node an-a01dr01: done [2026-11-19T09:00:35.138Z] Thu Nov 19 04:00:35 EST 2026 actual: 4 expected: 5 [2026-11-19T09:00:46.119Z] cloud-init status node an-striker01: done [2026-11-19T09:00:46.852Z] cloud-init status node an-striker02: done [2026-11-19T09:00:48.039Z] cloud-init status node an-a01n01: done [2026-11-19T09:00:48.774Z] cloud-init status node an-a01n02: done [2026-11-19T09:00:49.509Z] cloud-init status node an-a01dr01: done [2026-11-19T09:00:49.509Z] Thu Nov 19 04:00:49 EST 2026 actual: 5 expected: 5 [2026-11-19T09:00:49.509Z] Building simengine model [2026-11-19T09:01:02.356Z] Generating anvil striker auto config [2026-11-19T09:01:04.677Z] Cloning into 'anvil-test-suite'... [2026-11-19T09:01:05.021Z] generated ansible config: [2026-11-19T09:01:05.021Z] ========================= [2026-11-19T09:01:05.021Z] [defaults] [2026-11-19T09:01:05.021Z] inventory = ansible-inventory [2026-11-19T09:01:05.021Z] forks = 500 [2026-11-19T09:01:05.021Z] any_errors_fatal = true [2026-11-19T09:01:05.021Z] display_args_to_stdout = true [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:05.021Z] [ssh_connection] [2026-11-19T09:01:05.021Z] retries = 20 [2026-11-19T09:01:05.021Z] ========================= [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:05.021Z] generated ansible inventory: [2026-11-19T09:01:05.021Z] ============================ [2026-11-19T09:01:05.021Z] [strikers] [2026-11-19T09:01:05.021Z] an-striker01 [2026-11-19T09:01:05.021Z] an-striker02 [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:05.021Z] [nodes] [2026-11-19T09:01:05.021Z] an-a01n01 [2026-11-19T09:01:05.021Z] an-a01n02 [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:05.021Z] [drnodes] [2026-11-19T09:01:05.021Z] ============================ [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:05.021Z] Checking connectivity to nodes [2026-11-19T09:01:05.021Z] [2026-11-19T09:01:07.340Z] an-a01n01 | SUCCESS => { [2026-11-19T09:01:07.340Z] "ansible_facts": { [2026-11-19T09:01:07.340Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-11-19T09:01:07.340Z] }, [2026-11-19T09:01:07.340Z] "changed": false, [2026-11-19T09:01:07.340Z] "ping": "pong" [2026-11-19T09:01:07.340Z] } [2026-11-19T09:01:07.340Z] an-a01n02 | SUCCESS => { [2026-11-19T09:01:07.340Z] "ansible_facts": { [2026-11-19T09:01:07.340Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-11-19T09:01:07.340Z] }, [2026-11-19T09:01:07.340Z] "changed": false, [2026-11-19T09:01:07.340Z] "ping": "pong" [2026-11-19T09:01:07.340Z] } [2026-11-19T09:01:07.340Z] an-striker01 | SUCCESS => { [2026-11-19T09:01:07.340Z] "ansible_facts": { [2026-11-19T09:01:07.340Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-11-19T09:01:07.340Z] }, [2026-11-19T09:01:07.340Z] "changed": false, [2026-11-19T09:01:07.340Z] "ping": "pong" [2026-11-19T09:01:07.340Z] } [2026-11-19T09:01:07.340Z] an-striker02 | SUCCESS => { [2026-11-19T09:01:07.340Z] "ansible_facts": { [2026-11-19T09:01:07.340Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-11-19T09:01:07.340Z] }, [2026-11-19T09:01:07.340Z] "changed": false, [2026-11-19T09:01:07.340Z] "ping": "pong" [2026-11-19T09:01:07.340Z] } [2026-11-19T09:01:07.340Z] [2026-11-19T09:01:08.072Z] [2026-11-19T09:01:08.072Z] PLAY [Setup all nodes] ********************************************************* [2026-11-19T09:01:08.072Z] [2026-11-19T09:01:08.072Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:01:10.393Z] ok: [an-a01n01] [2026-11-19T09:01:10.393Z] ok: [an-a01n02] [2026-11-19T09:01:10.393Z] ok: [an-striker02] [2026-11-19T09:01:10.393Z] ok: [an-striker01] [2026-11-19T09:01:10.393Z] [2026-11-19T09:01:10.393Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-11-19T09:01:11.126Z] changed: [an-a01n01] [2026-11-19T09:01:11.126Z] changed: [an-a01n02] [2026-11-19T09:01:11.126Z] changed: [an-striker01] [2026-11-19T09:01:11.126Z] changed: [an-striker02] [2026-11-19T09:01:11.126Z] [2026-11-19T09:01:11.126Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-11-19T09:01:11.859Z] ok: [an-a01n02] [2026-11-19T09:01:11.859Z] ok: [an-a01n01] [2026-11-19T09:01:11.859Z] ok: [an-striker01] [2026-11-19T09:01:11.859Z] ok: [an-striker02] [2026-11-19T09:01:11.859Z] [2026-11-19T09:01:11.859Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-11-19T09:01:13.046Z] changed: [an-a01n02] [2026-11-19T09:01:13.046Z] changed: [an-a01n01] [2026-11-19T09:01:13.046Z] changed: [an-striker02] [2026-11-19T09:01:13.046Z] changed: [an-striker01] [2026-11-19T09:01:13.046Z] [2026-11-19T09:01:13.046Z] PLAY [setup strikers] ********************************************************** [2026-11-19T09:01:13.046Z] [2026-11-19T09:01:13.046Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:01:14.756Z] ok: [an-striker02] [2026-11-19T09:01:14.756Z] ok: [an-striker01] [2026-11-19T09:01:14.756Z] [2026-11-19T09:01:14.756Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-11-19T09:01:16.466Z] changed: [an-striker02] [2026-11-19T09:01:16.466Z] changed: [an-striker01] [2026-11-19T09:01:16.466Z] [2026-11-19T09:01:16.466Z] PLAY [setup lvm storage] ******************************************************* [2026-11-19T09:01:16.466Z] [2026-11-19T09:01:16.466Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:01:18.176Z] ok: [an-a01n01] [2026-11-19T09:01:18.176Z] ok: [an-a01n02] [2026-11-19T09:01:18.176Z] [2026-11-19T09:01:18.176Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-11-19T09:01:18.176Z] ] **************** [2026-11-19T09:01:18.912Z] ok: [an-a01n01] [2026-11-19T09:01:18.912Z] ok: [an-a01n02] [2026-11-19T09:01:18.912Z] [2026-11-19T09:01:18.912Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-11-19T09:01:18.912Z] skipping: [an-a01n01] [2026-11-19T09:01:18.912Z] [2026-11-19T09:01:18.912Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-11-19T09:01:20.096Z] changed: [an-a01n02] [2026-11-19T09:01:20.096Z] changed: [an-a01n01] [2026-11-19T09:01:20.096Z] [2026-11-19T09:01:20.096Z] PLAY [deploy anvil] ************************************************************ [2026-11-19T09:01:20.096Z] [2026-11-19T09:01:20.096Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:01:21.285Z] ok: [an-striker02] [2026-11-19T09:01:21.285Z] ok: [an-striker01] [2026-11-19T09:01:21.285Z] [2026-11-19T09:01:21.285Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-11-19T09:01:22.470Z] ok: [an-striker02] [2026-11-19T09:01:22.470Z] ok: [an-striker01] [2026-11-19T09:01:22.470Z] [2026-11-19T09:01:22.470Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-11-19T09:01:23.656Z] changed: [an-striker02] [2026-11-19T09:01:23.656Z] changed: [an-striker01] [2026-11-19T09:01:23.656Z] [2026-11-19T09:01:23.656Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-11-19T09:01:23.656Z] ] *** [2026-11-19T09:01:25.972Z] changed: [an-striker02] [2026-11-19T09:01:25.972Z] changed: [an-striker01] [2026-11-19T09:01:25.972Z] [2026-11-19T09:01:25.972Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:01:25.972Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:01:25.972Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-11-19T09:01:25.972Z] ] *** [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-11-19T09:06:50.799Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-11-19T09:06:50.799Z] ok: [an-striker02 -> localhost] [2026-11-19T09:06:50.799Z] ok: [an-striker01 -> localhost] [2026-11-19T09:06:50.799Z] [2026-11-19T09:06:50.799Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-11-19T09:06:50.799Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-11-19T09:06:50.799Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-11-19T09:06:50.799Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-11-19T09:06:50.799Z] echo "DB has completed sync" [2026-11-19T09:06:50.799Z] exit 0 [2026-11-19T09:06:50.799Z] fi [2026-11-19T09:06:50.799Z] if [ "$dbcon" != 2 ]; then [2026-11-19T09:06:50.799Z] echo "Detected $dbcon db connections (expected 2)" [2026-11-19T09:06:50.799Z] exit 1 [2026-11-19T09:06:50.799Z] fi [2026-11-19T09:06:50.799Z] ] *** [2026-11-19T09:07:18.777Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-11-19T09:07:18.777Z] ok: [an-striker01] [2026-11-19T09:07:18.777Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-11-19T09:07:18.777Z] ok: [an-striker02] [2026-11-19T09:07:18.777Z] [2026-11-19T09:07:18.777Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:07:18.777Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:07:18.777Z] --username=postgres anvil -t \ [2026-11-19T09:07:18.777Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-11-19T09:07:18.777Z] grep altee | \ [2026-11-19T09:07:18.777Z] grep 1$ | \ [2026-11-19T09:07:18.777Z] wc -l [2026-11-19T09:07:18.777Z] ] *** [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-11-19T09:12:58.693Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-11-19T09:12:58.694Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (68 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (67 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (66 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left). [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (65 retries left). [2026-11-19T09:17:08.032Z] ok: [an-striker01 -> localhost] [2026-11-19T09:17:08.032Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (64 retries left). [2026-11-19T09:17:08.032Z] ok: [an-striker02 -> localhost] [2026-11-19T09:17:08.032Z] [2026-11-19T09:17:08.032Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:17:08.032Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:17:08.032Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-11-19T09:17:08.032Z] grep altee | \ [2026-11-19T09:17:08.032Z] wc -l [2026-11-19T09:17:08.032Z] ] *** [2026-11-19T09:17:08.032Z] ok: [an-striker01 -> localhost] [2026-11-19T09:17:08.032Z] ok: [an-striker02 -> localhost] [2026-11-19T09:17:08.032Z] [2026-11-19T09:17:08.032Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:17:08.032Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:17:08.032Z] --username=postgres anvil -t \ [2026-11-19T09:17:08.032Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-11-19T09:17:08.032Z] grep altee | \ [2026-11-19T09:17:08.032Z] grep 1$ | \ [2026-11-19T09:17:08.032Z] wc -l [2026-11-19T09:17:08.032Z] ] *** [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-11-19T09:19:16.677Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-11-19T09:19:16.677Z] ok: [an-striker02 -> localhost] [2026-11-19T09:19:16.677Z] ok: [an-striker01 -> localhost] [2026-11-19T09:19:16.677Z] [2026-11-19T09:19:16.677Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:19:16.677Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:19:16.677Z] --username=postgres anvil -t \ [2026-11-19T09:19:16.677Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-11-19T09:19:16.677Z] grep altee | \ [2026-11-19T09:19:16.677Z] grep 1$ | \ [2026-11-19T09:19:16.677Z] wc -l [2026-11-19T09:19:16.677Z] ] *** [2026-11-19T09:19:29.967Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-11-19T09:19:29.967Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-11-19T09:19:29.967Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-11-19T09:19:29.967Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-11-19T09:19:29.967Z] ok: [an-striker02 -> localhost] [2026-11-19T09:19:29.967Z] ok: [an-striker01 -> localhost] [2026-11-19T09:19:29.967Z] [2026-11-19T09:19:29.967Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-11-19T09:19:29.967Z] psql --host="{{ inventory_hostname }}" \ [2026-11-19T09:19:29.967Z] --username=postgres anvil -t \ [2026-11-19T09:19:29.967Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-11-19T09:19:29.967Z] grep altee | \ [2026-11-19T09:19:29.967Z] grep 1$ | \ [2026-11-19T09:19:29.967Z] wc -l [2026-11-19T09:19:29.967Z] ] *** [2026-11-19T09:19:29.967Z] ok: [an-striker01 -> localhost] [2026-11-19T09:19:29.967Z] ok: [an-striker02 -> localhost] [2026-11-19T09:19:29.967Z] [2026-11-19T09:19:29.967Z] PLAY [check network status after deployment] *********************************** [2026-11-19T09:19:29.967Z] [2026-11-19T09:19:29.967Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:19:33.286Z] ok: [an-striker01] [2026-11-19T09:19:33.286Z] ok: [an-a01n02] [2026-11-19T09:19:33.286Z] ok: [an-a01n01] [2026-11-19T09:19:33.286Z] ok: [an-striker02] [2026-11-19T09:19:33.286Z] [2026-11-19T09:19:33.286Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-11-19T09:19:33.286Z] if [ "$device" = "" ]; then [2026-11-19T09:19:33.286Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-11-19T09:19:33.286Z] exit 1 [2026-11-19T09:19:33.286Z] fi [2026-11-19T09:19:33.287Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-11-19T09:19:33.287Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-11-19T09:19:33.287Z] exit 1 [2026-11-19T09:19:33.287Z] fi [2026-11-19T09:19:33.287Z] exit 0 [2026-11-19T09:19:33.287Z] ] *** [2026-11-19T09:19:41.068Z] skipping: [an-a01n01] => (item=lo) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=sn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=ifn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=ifn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-11-19T09:19:41.068Z] skipping: [an-striker02] => (item=lo) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=sn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=bcn1_link1) [2026-11-19T09:19:41.068Z] skipping: [an-striker01] => (item=lo) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=sn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=sn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=bcn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=ifn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-11-19T09:19:41.068Z] skipping: [an-a01n02] => (item=lo) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=bcn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-striker01] => (item=ifn1_link2) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-11-19T09:19:41.068Z] changed: [an-striker02] => (item=bcn1_link1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-11-19T09:19:41.068Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-11-19T09:19:41.068Z] [2026-11-19T09:19:41.068Z] PLAY [check pacemaker status after deployment] ********************************* [2026-11-19T09:19:41.068Z] [2026-11-19T09:19:41.068Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:19:42.252Z] ok: [an-a01n01] [2026-11-19T09:19:42.252Z] ok: [an-a01n02] [2026-11-19T09:19:42.252Z] [2026-11-19T09:19:42.252Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-11-19T09:19:42.252Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T09:19:42.252Z] [2026-11-19T09:19:42.252Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T09:19:42.252Z] ] *** [2026-11-19T09:19:46.119Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T09:19:46.119Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T09:19:46.119Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T09:19:46.119Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T09:19:46.119Z] [2026-11-19T09:19:46.119Z] PLAY [check drbd kmod] ********************************************************* [2026-11-19T09:19:46.119Z] [2026-11-19T09:19:46.119Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:19:47.305Z] ok: [an-a01n01] [2026-11-19T09:19:47.305Z] ok: [an-a01n02] [2026-11-19T09:19:47.305Z] [2026-11-19T09:19:47.305Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-11-19T09:19:47.305Z] ] *** [2026-11-19T09:19:48.038Z] ok: [an-a01n02] [2026-11-19T09:19:48.038Z] ok: [an-a01n01] [2026-11-19T09:19:48.038Z] [2026-11-19T09:19:48.038Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-11-19T09:19:48.038Z] ] *********************** [2026-11-19T09:19:48.038Z] skipping: [an-a01n01] [2026-11-19T09:19:48.038Z] skipping: [an-a01n02] [2026-11-19T09:19:48.038Z] [2026-11-19T09:19:48.038Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-11-19T09:19:48.038Z] ] *** [2026-11-19T09:19:48.379Z] skipping: [an-a01n01] [2026-11-19T09:19:48.379Z] skipping: [an-a01n02] [2026-11-19T09:19:48.379Z] [2026-11-19T09:19:48.379Z] PLAY [Copy iso images to striker01] ******************************************** [2026-11-19T09:19:48.379Z] [2026-11-19T09:19:48.379Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:19:50.087Z] ok: [an-striker01] [2026-11-19T09:19:50.087Z] [2026-11-19T09:19:50.087Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-11-19T09:19:50.821Z] ok: [an-striker01] [2026-11-19T09:19:50.821Z] [2026-11-19T09:19:50.821Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-11-19T09:19:51.552Z] changed: [an-striker01] [2026-11-19T09:19:51.552Z] [2026-11-19T09:19:51.552Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-11-19T09:19:58.945Z] changed: [an-striker01] => (item=deploy1.iso) [2026-11-19T09:19:58.945Z] changed: [an-striker01] => (item=deploy2.iso) [2026-11-19T09:19:58.945Z] changed: [an-striker01] => (item=deploy3.iso) [2026-11-19T09:19:58.945Z] changed: [an-striker01] => (item=deploy4.iso) [2026-11-19T09:19:58.945Z] changed: [an-striker01] => (item=deploy5.iso) [2026-11-19T09:19:58.945Z] [2026-11-19T09:19:58.945Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-11-19T09:19:58.945Z] ] *** [2026-11-19T09:20:00.130Z] changed: [an-striker01] [2026-11-19T09:20:00.130Z] [2026-11-19T09:20:00.130Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-11-19T09:20:00.863Z] changed: [an-striker01] [2026-11-19T09:20:00.863Z] [2026-11-19T09:20:00.863Z] PLAY [Check iso images propagation] ******************************************** [2026-11-19T09:20:00.863Z] [2026-11-19T09:20:00.863Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:20:03.181Z] ok: [an-a01n02] [2026-11-19T09:20:03.181Z] ok: [an-a01n01] [2026-11-19T09:20:03.181Z] ok: [an-striker01] [2026-11-19T09:20:03.181Z] ok: [an-striker02] [2026-11-19T09:20:03.181Z] [2026-11-19T09:20:03.181Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-11-19T09:20:03.181Z] exit 1 [2026-11-19T09:20:03.181Z] else [2026-11-19T09:20:03.181Z] exit 0 [2026-11-19T09:20:03.181Z] fi [2026-11-19T09:20:03.181Z] ] *** [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-11-19T09:21:41.615Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-11-19T09:21:41.615Z] ok: [an-striker01] [2026-11-19T09:21:41.615Z] ok: [an-striker02] [2026-11-19T09:21:41.615Z] ok: [an-a01n01] [2026-11-19T09:21:41.615Z] ok: [an-a01n02] [2026-11-19T09:21:41.615Z] [2026-11-19T09:21:41.615Z] PLAY [example tests] *********************************************************** [2026-11-19T09:21:41.615Z] [2026-11-19T09:21:41.615Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:21:41.615Z] ok: [an-a01n02] [2026-11-19T09:21:41.615Z] ok: [an-a01n01] [2026-11-19T09:21:41.615Z] ok: [an-striker02] [2026-11-19T09:21:41.615Z] ok: [an-striker01] [2026-11-19T09:21:41.615Z] [2026-11-19T09:21:41.615Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-11-19T09:21:41.615Z] ] *** [2026-11-19T09:21:41.956Z] skipping: [an-a01n01] [2026-11-19T09:21:41.956Z] skipping: [an-a01n02] [2026-11-19T09:21:41.956Z] ok: [an-striker01] [2026-11-19T09:21:41.956Z] ok: [an-striker02] [2026-11-19T09:21:41.956Z] [2026-11-19T09:21:41.956Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-11-19T09:21:41.956Z] ] *** [2026-11-19T09:21:41.956Z] skipping: [an-striker01] [2026-11-19T09:21:41.956Z] skipping: [an-striker02] [2026-11-19T09:21:42.690Z] ok: [an-a01n01] [2026-11-19T09:21:42.690Z] ok: [an-a01n02] [2026-11-19T09:21:42.690Z] [2026-11-19T09:21:42.690Z] TASK [inject failure _raw_params=/usr/bin/false [2026-11-19T09:21:42.690Z] ] ******************************* [2026-11-19T09:21:42.690Z] skipping: [an-striker01] [2026-11-19T09:21:42.690Z] skipping: [an-striker02] [2026-11-19T09:21:43.032Z] skipping: [an-a01n01] [2026-11-19T09:21:43.032Z] skipping: [an-a01n02] [2026-11-19T09:21:43.032Z] [2026-11-19T09:21:43.032Z] PLAY [Record anvil available resources from striker01] ************************* [2026-11-19T09:21:43.032Z] [2026-11-19T09:21:43.032Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:21:44.739Z] ok: [an-striker01] [2026-11-19T09:21:44.739Z] [2026-11-19T09:21:44.739Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-11-19T09:21:44.739Z] ] *** [2026-11-19T09:21:48.607Z] ok: [an-striker01] [2026-11-19T09:21:48.607Z] [2026-11-19T09:21:48.607Z] PLAY [check selinux status] **************************************************** [2026-11-19T09:21:48.607Z] [2026-11-19T09:21:48.607Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:21:50.314Z] ok: [an-a01n01] [2026-11-19T09:21:50.314Z] ok: [an-a01n02] [2026-11-19T09:21:50.314Z] ok: [an-striker01] [2026-11-19T09:21:50.314Z] ok: [an-striker02] [2026-11-19T09:21:50.314Z] [2026-11-19T09:21:50.314Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-11-19T09:21:50.314Z] skipping: [an-striker01] [2026-11-19T09:21:50.314Z] skipping: [an-striker02] [2026-11-19T09:21:50.314Z] skipping: [an-a01n01] [2026-11-19T09:21:50.314Z] skipping: [an-a01n02] [2026-11-19T09:21:50.314Z] [2026-11-19T09:21:50.314Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-11-19T09:21:50.656Z] ok: [an-striker01] => { [2026-11-19T09:21:50.656Z] "ansible_selinux": { [2026-11-19T09:21:50.656Z] "config_mode": "enforcing", [2026-11-19T09:21:50.656Z] "mode": "enforcing", [2026-11-19T09:21:50.656Z] "policyvers": 33, [2026-11-19T09:21:50.656Z] "status": "enabled", [2026-11-19T09:21:50.656Z] "type": "targeted" [2026-11-19T09:21:50.656Z] }, [2026-11-19T09:21:50.656Z] "failed_when_result": false [2026-11-19T09:21:50.656Z] } [2026-11-19T09:21:50.656Z] ok: [an-striker02] => { [2026-11-19T09:21:50.656Z] "ansible_selinux": { [2026-11-19T09:21:50.656Z] "config_mode": "enforcing", [2026-11-19T09:21:50.656Z] "mode": "enforcing", [2026-11-19T09:21:50.656Z] "policyvers": 33, [2026-11-19T09:21:50.656Z] "status": "enabled", [2026-11-19T09:21:50.656Z] "type": "targeted" [2026-11-19T09:21:50.656Z] }, [2026-11-19T09:21:50.656Z] "failed_when_result": false [2026-11-19T09:21:50.656Z] } [2026-11-19T09:21:50.656Z] ok: [an-a01n01] => { [2026-11-19T09:21:50.656Z] "ansible_selinux": { [2026-11-19T09:21:50.656Z] "config_mode": "enforcing", [2026-11-19T09:21:50.656Z] "mode": "enforcing", [2026-11-19T09:21:50.656Z] "policyvers": 33, [2026-11-19T09:21:50.656Z] "status": "enabled", [2026-11-19T09:21:50.656Z] "type": "targeted" [2026-11-19T09:21:50.656Z] }, [2026-11-19T09:21:50.656Z] "failed_when_result": false [2026-11-19T09:21:50.656Z] } [2026-11-19T09:21:50.656Z] ok: [an-a01n02] => { [2026-11-19T09:21:50.656Z] "ansible_selinux": { [2026-11-19T09:21:50.656Z] "config_mode": "enforcing", [2026-11-19T09:21:50.656Z] "mode": "enforcing", [2026-11-19T09:21:50.656Z] "policyvers": 33, [2026-11-19T09:21:50.656Z] "status": "enabled", [2026-11-19T09:21:50.656Z] "type": "targeted" [2026-11-19T09:21:50.656Z] }, [2026-11-19T09:21:50.656Z] "failed_when_result": false [2026-11-19T09:21:50.656Z] } [2026-11-19T09:21:50.656Z] [2026-11-19T09:21:50.656Z] PLAY [Create random file on striker01] ***************************************** [2026-11-19T09:21:50.656Z] [2026-11-19T09:21:50.656Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:21:52.362Z] ok: [an-striker01] [2026-11-19T09:21:52.362Z] [2026-11-19T09:21:52.362Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-11-19T09:21:54.071Z] changed: [an-striker01] [2026-11-19T09:21:54.071Z] [2026-11-19T09:21:54.071Z] PLAY [Check archive file propagation] ****************************************** [2026-11-19T09:21:54.071Z] [2026-11-19T09:21:54.071Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:21:56.391Z] ok: [an-a01n01] [2026-11-19T09:21:56.391Z] ok: [an-striker02] [2026-11-19T09:21:56.391Z] ok: [an-a01n02] [2026-11-19T09:21:56.391Z] ok: [an-striker01] [2026-11-19T09:21:56.391Z] [2026-11-19T09:21:56.391Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-11-19T09:21:56.391Z] exit 1 [2026-11-19T09:21:56.391Z] else [2026-11-19T09:21:56.391Z] exit 0 [2026-11-19T09:21:56.391Z] fi [2026-11-19T09:21:56.391Z] ] *** [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-11-19T09:23:19.765Z] ok: [an-striker01] [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-11-19T09:23:19.765Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-11-19T09:23:19.766Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-11-19T09:23:19.766Z] ok: [an-a01n01] [2026-11-19T09:23:19.766Z] ok: [an-a01n02] [2026-11-19T09:23:19.766Z] ok: [an-striker02] [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] PLAY [test fencing] ************************************************************ [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:23:19.766Z] ok: [an-a01n01] [2026-11-19T09:23:19.766Z] ok: [an-a01n02] [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T09:23:19.766Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T09:23:19.766Z] ] *** [2026-11-19T09:23:19.766Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T09:23:19.766Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T09:23:19.766Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T09:23:19.766Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-11-19T09:23:19.766Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T09:23:19.766Z] [2026-11-19T09:23:19.766Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T09:23:19.766Z] ] ********** [2026-11-19T09:23:20.108Z] skipping: [an-a01n02] [2026-11-19T09:23:20.108Z] changed: [an-a01n01] [2026-11-19T09:23:20.108Z] [2026-11-19T09:23:20.108Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T09:23:20.452Z] skipping: [an-a01n02] [2026-11-19T09:23:20.452Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-11-19T09:23:20.452Z] [2026-11-19T09:23:20.452Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T09:26:44.548Z] ok: [an-a01n01 -> localhost] [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T09:26:44.548Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T09:26:44.548Z] ] *** [2026-11-19T09:26:44.548Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T09:26:44.548Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T09:26:44.548Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T09:26:44.548Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-11-19T09:26:44.548Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T09:26:44.548Z] ] ********** [2026-11-19T09:26:44.548Z] skipping: [an-a01n01] [2026-11-19T09:26:44.548Z] changed: [an-a01n02] [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T09:26:44.548Z] skipping: [an-a01n01] [2026-11-19T09:26:44.548Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-11-19T09:26:44.548Z] [2026-11-19T09:26:44.548Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T09:29:53.564Z] ok: [an-a01n02 -> localhost] [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T09:29:53.564Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T09:29:53.564Z] ] *** [2026-11-19T09:29:53.564Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T09:29:53.564Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T09:29:53.564Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T09:29:53.564Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] PLAY [advanced server tests] *************************************************** [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T09:29:53.564Z] ok: [an-a01n02] [2026-11-19T09:29:53.564Z] ok: [an-a01n01] [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T09:29:53.564Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T09:29:53.564Z] [2026-11-19T09:29:53.564Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T09:29:53.564Z] ] *** [2026-11-19T09:29:55.887Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T09:29:55.887Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T09:29:55.887Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T09:29:55.887Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T09:29:55.887Z] [2026-11-19T09:29:55.887Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-11-19T09:29:55.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-11-19T09:29:55.887Z] [2026-11-19T09:29:55.887Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-11-19T09:29:55.887Z] if [ -z "$storage" ]; then [2026-11-19T09:29:55.887Z] echo "Unable to detect shared storage" [2026-11-19T09:29:55.887Z] exit 1 [2026-11-19T09:29:55.887Z] fi [2026-11-19T09:29:55.887Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T09:29:55.887Z] ] *** [2026-11-19T09:30:00.746Z] changed: [an-a01n01] [2026-11-19T09:30:00.746Z] changed: [an-a01n02] [2026-11-19T09:30:00.746Z] [2026-11-19T09:30:00.746Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-11-19T09:30:00.746Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-11-19T09:30:00.746Z] [2026-11-19T09:30:00.746Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-11-19T09:30:00.746Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-11-19T09:30:00.746Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-11-19T09:30:00.746Z] echo "Unable to detect all deploy iso" [2026-11-19T09:30:00.746Z] echo "Detected isos" [2026-11-19T09:30:00.746Z] echo "$isos" [2026-11-19T09:30:00.746Z] exit 1 [2026-11-19T09:30:00.746Z] fi [2026-11-19T09:30:00.746Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-11-19T09:30:00.746Z] if [ -z "$installiso" ]; then [2026-11-19T09:30:00.746Z] echo "Unable to detect install iso" [2026-11-19T09:30:00.746Z] echo "Detected isos" [2026-11-19T09:30:00.746Z] echo "$isos" [2026-11-19T09:30:00.746Z] exit 1 [2026-11-19T09:30:00.746Z] fi [2026-11-19T09:30:00.746Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T09:30:00.746Z] ] *** [2026-11-19T09:30:05.601Z] changed: [an-a01n02] [2026-11-19T09:30:05.601Z] changed: [an-a01n01] [2026-11-19T09:30:05.601Z] [2026-11-19T09:30:05.601Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-11-19T09:30:05.601Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:30:05.601Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:30:05.601Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:30:05.601Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:30:05.601Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:30:05.601Z] [2026-11-19T09:30:05.601Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T09:30:05.601Z] ] *** [2026-11-19T09:30:06.790Z] skipping: [an-a01n02] [2026-11-19T09:30:06.790Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.829241", "end": "2026-11-19 04:30:06.634873", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 04:30:05.805632", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T09:30:06.790Z] ...ignoring [2026-11-19T09:30:06.790Z] [2026-11-19T09:30:06.790Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T09:30:06.790Z] --ci-test \ [2026-11-19T09:30:06.790Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T09:30:06.790Z] --os almalinux9 \ [2026-11-19T09:30:06.790Z] --cpu 2 --ram 4G \ [2026-11-19T09:30:06.790Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T09:30:06.790Z] --install-media {{ installisouuid }} \ [2026-11-19T09:30:06.790Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T09:30:06.790Z] ] *** [2026-11-19T09:30:15.830Z] skipping: [an-a01n02] [2026-11-19T09:30:15.830Z] changed: [an-a01n01] [2026-11-19T09:30:15.830Z] [2026-11-19T09:30:15.830Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T09:30:15.830Z] skipping: [an-a01n01] [2026-11-19T09:30:15.830Z] skipping: [an-a01n02] [2026-11-19T09:30:15.830Z] [2026-11-19T09:30:15.830Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T09:30:15.830Z] skipping: [an-a01n01] [2026-11-19T09:30:15.830Z] skipping: [an-a01n02] [2026-11-19T09:30:15.830Z] [2026-11-19T09:30:15.830Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T09:30:15.830Z] skipping: [an-a01n01] [2026-11-19T09:30:15.830Z] skipping: [an-a01n02] [2026-11-19T09:30:15.830Z] [2026-11-19T09:30:15.830Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T09:30:15.830Z] skipping: [an-a01n01] [2026-11-19T09:30:15.830Z] skipping: [an-a01n02] [2026-11-19T09:30:15.830Z] [2026-11-19T09:30:15.830Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T09:30:15.830Z] ] *** [2026-11-19T09:30:17.019Z] skipping: [an-a01n02] [2026-11-19T09:30:17.019Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.896816", "end": "2026-11-19 04:30:16.651023", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 04:30:15.754207", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T09:30:17.019Z] ...ignoring [2026-11-19T09:30:17.019Z] [2026-11-19T09:30:17.019Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T09:30:17.019Z] --ci-test \ [2026-11-19T09:30:17.019Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T09:30:17.019Z] --os almalinux9 \ [2026-11-19T09:30:17.019Z] --cpu 2 --ram 4G \ [2026-11-19T09:30:17.019Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T09:30:17.019Z] --install-media {{ installisouuid }} \ [2026-11-19T09:30:17.019Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T09:30:17.019Z] ] *** [2026-11-19T09:30:26.218Z] skipping: [an-a01n02] [2026-11-19T09:30:26.218Z] changed: [an-a01n01] [2026-11-19T09:30:26.218Z] [2026-11-19T09:30:26.218Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T09:30:26.218Z] skipping: [an-a01n01] [2026-11-19T09:30:26.218Z] skipping: [an-a01n02] [2026-11-19T09:30:26.218Z] [2026-11-19T09:30:26.218Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T09:30:26.218Z] skipping: [an-a01n01] [2026-11-19T09:30:26.218Z] skipping: [an-a01n02] [2026-11-19T09:30:26.218Z] [2026-11-19T09:30:26.218Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T09:30:26.218Z] skipping: [an-a01n01] [2026-11-19T09:30:26.218Z] skipping: [an-a01n02] [2026-11-19T09:30:26.218Z] [2026-11-19T09:30:26.218Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T09:30:26.218Z] skipping: [an-a01n01] [2026-11-19T09:30:26.218Z] skipping: [an-a01n02] [2026-11-19T09:30:26.218Z] [2026-11-19T09:30:26.218Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T09:30:26.218Z] ] *** [2026-11-19T09:30:27.304Z] skipping: [an-a01n02] [2026-11-19T09:30:27.304Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.953567", "end": "2026-11-19 04:30:26.962022", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 04:30:26.008455", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T09:30:27.304Z] ...ignoring [2026-11-19T09:30:27.304Z] [2026-11-19T09:30:27.304Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T09:30:27.304Z] --ci-test \ [2026-11-19T09:30:27.304Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T09:30:27.304Z] --os almalinux9 \ [2026-11-19T09:30:27.304Z] --cpu 2 --ram 4G \ [2026-11-19T09:30:27.304Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T09:30:27.304Z] --install-media {{ installisouuid }} \ [2026-11-19T09:30:27.304Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T09:30:27.304Z] ] *** [2026-11-19T09:30:36.341Z] skipping: [an-a01n02] [2026-11-19T09:30:36.341Z] changed: [an-a01n01] [2026-11-19T09:30:36.341Z] [2026-11-19T09:30:36.341Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T09:30:36.341Z] skipping: [an-a01n01] [2026-11-19T09:30:36.341Z] skipping: [an-a01n02] [2026-11-19T09:30:36.341Z] [2026-11-19T09:30:36.341Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T09:30:36.684Z] skipping: [an-a01n01] [2026-11-19T09:30:36.684Z] skipping: [an-a01n02] [2026-11-19T09:30:36.684Z] [2026-11-19T09:30:36.684Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T09:30:36.684Z] skipping: [an-a01n01] [2026-11-19T09:30:36.684Z] skipping: [an-a01n02] [2026-11-19T09:30:36.684Z] [2026-11-19T09:30:36.684Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T09:30:36.684Z] skipping: [an-a01n01] [2026-11-19T09:30:36.684Z] skipping: [an-a01n02] [2026-11-19T09:30:36.684Z] [2026-11-19T09:30:36.684Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T09:30:36.684Z] ] *** [2026-11-19T09:30:38.396Z] skipping: [an-a01n02] [2026-11-19T09:30:38.396Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.906946", "end": "2026-11-19 04:30:37.905167", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 04:30:36.998221", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T09:30:38.396Z] ...ignoring [2026-11-19T09:30:38.396Z] [2026-11-19T09:30:38.396Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T09:30:38.396Z] --ci-test \ [2026-11-19T09:30:38.396Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T09:30:38.396Z] --os almalinux9 \ [2026-11-19T09:30:38.396Z] --cpu 2 --ram 4G \ [2026-11-19T09:30:38.396Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T09:30:38.396Z] --install-media {{ installisouuid }} \ [2026-11-19T09:30:38.396Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T09:30:38.396Z] ] *** [2026-11-19T09:30:49.377Z] skipping: [an-a01n02] [2026-11-19T09:30:49.377Z] changed: [an-a01n01] [2026-11-19T09:30:49.377Z] [2026-11-19T09:30:49.377Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T09:30:49.377Z] skipping: [an-a01n01] [2026-11-19T09:30:49.377Z] skipping: [an-a01n02] [2026-11-19T09:30:49.377Z] [2026-11-19T09:30:49.377Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T09:30:49.377Z] skipping: [an-a01n01] [2026-11-19T09:30:49.377Z] skipping: [an-a01n02] [2026-11-19T09:30:49.377Z] [2026-11-19T09:30:49.377Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T09:30:49.377Z] skipping: [an-a01n01] [2026-11-19T09:30:49.377Z] skipping: [an-a01n02] [2026-11-19T09:30:49.377Z] [2026-11-19T09:30:49.377Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T09:30:49.377Z] skipping: [an-a01n01] [2026-11-19T09:30:49.377Z] skipping: [an-a01n02] [2026-11-19T09:30:49.377Z] [2026-11-19T09:30:49.377Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T09:30:49.377Z] ] *** [2026-11-19T09:30:50.568Z] skipping: [an-a01n02] [2026-11-19T09:30:50.568Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.162176", "end": "2026-11-19 04:30:50.144630", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 04:30:48.982454", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T09:30:50.568Z] ...ignoring [2026-11-19T09:30:50.568Z] [2026-11-19T09:30:50.568Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T09:30:50.568Z] --ci-test \ [2026-11-19T09:30:50.568Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T09:30:50.568Z] --os almalinux9 \ [2026-11-19T09:30:50.568Z] --cpu 2 --ram 4G \ [2026-11-19T09:30:50.568Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T09:30:50.568Z] --install-media {{ installisouuid }} \ [2026-11-19T09:30:50.568Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T09:30:50.568Z] ] *** [2026-11-19T09:31:01.554Z] skipping: [an-a01n02] [2026-11-19T09:31:01.554Z] changed: [an-a01n01] [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T09:31:01.554Z] skipping: [an-a01n01] [2026-11-19T09:31:01.554Z] skipping: [an-a01n02] [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T09:31:01.554Z] skipping: [an-a01n01] [2026-11-19T09:31:01.554Z] skipping: [an-a01n02] [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T09:31:01.554Z] skipping: [an-a01n01] [2026-11-19T09:31:01.554Z] skipping: [an-a01n02] [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T09:31:01.554Z] skipping: [an-a01n01] [2026-11-19T09:31:01.554Z] skipping: [an-a01n02] [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-11-19T09:31:01.554Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:31:01.554Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:31:01.554Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:31:01.554Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:31:01.554Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:31:01.554Z] [2026-11-19T09:31:01.554Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T09:31:01.554Z] ] *** [2026-11-19T09:31:29.534Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-11-19T09:31:29.534Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-11-19T09:31:29.534Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-11-19T09:31:29.534Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-11-19T09:31:29.534Z] ok: [an-a01n02] [2026-11-19T09:31:29.534Z] ok: [an-a01n01] [2026-11-19T09:31:29.534Z] [2026-11-19T09:31:29.534Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T09:31:29.534Z] ] *** [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-11-19T09:33:38.197Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-11-19T09:33:38.197Z] ok: [an-a01n01] [2026-11-19T09:33:38.197Z] ok: [an-a01n02] [2026-11-19T09:33:38.197Z] [2026-11-19T09:33:38.197Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T09:33:38.197Z] ] *** [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-11-19T09:35:31.737Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-11-19T09:35:31.737Z] ok: [an-a01n02] [2026-11-19T09:35:31.737Z] ok: [an-a01n01] [2026-11-19T09:35:31.737Z] [2026-11-19T09:35:31.737Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T09:35:31.737Z] ] *** [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-11-19T09:37:55.494Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-11-19T09:37:55.495Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-11-19T09:37:55.495Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-11-19T09:37:55.495Z] ok: [an-a01n02] [2026-11-19T09:37:55.495Z] ok: [an-a01n01] [2026-11-19T09:37:55.495Z] [2026-11-19T09:37:55.495Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T09:37:55.495Z] ] *** [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-11-19T09:40:49.442Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-11-19T09:40:49.442Z] ok: [an-a01n02] [2026-11-19T09:40:49.442Z] ok: [an-a01n01] [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-11-19T09:40:49.442Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:40:49.442Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:40:49.442Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:40:49.442Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:40:49.442Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T09:40:49.442Z] ] *** [2026-11-19T09:40:49.442Z] ok: [an-a01n02] [2026-11-19T09:40:49.442Z] ok: [an-a01n01] [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T09:40:49.442Z] ] *** [2026-11-19T09:40:49.442Z] ok: [an-a01n02] [2026-11-19T09:40:49.442Z] ok: [an-a01n01] [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T09:40:49.442Z] ] *** [2026-11-19T09:40:49.442Z] ok: [an-a01n02] [2026-11-19T09:40:49.442Z] ok: [an-a01n01] [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T09:40:49.442Z] ] *** [2026-11-19T09:40:49.442Z] ok: [an-a01n02] [2026-11-19T09:40:49.442Z] ok: [an-a01n01] [2026-11-19T09:40:49.442Z] [2026-11-19T09:40:49.442Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T09:40:49.442Z] ] *** [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-11-19T09:41:23.144Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-11-19T09:41:23.144Z] ok: [an-a01n02] [2026-11-19T09:41:23.144Z] ok: [an-a01n01] [2026-11-19T09:41:23.144Z] [2026-11-19T09:41:23.144Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-11-19T09:41:23.144Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:41:23.144Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:41:23.144Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:41:23.144Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:41:23.144Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:41:23.144Z] [2026-11-19T09:41:23.144Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T09:41:23.144Z] ] *** [2026-11-19T09:41:23.144Z] ok: [an-a01n02] [2026-11-19T09:41:23.144Z] ok: [an-a01n01] [2026-11-19T09:41:23.144Z] [2026-11-19T09:41:23.144Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T09:41:23.144Z] ] *** [2026-11-19T09:41:23.144Z] ok: [an-a01n02] [2026-11-19T09:41:23.144Z] ok: [an-a01n01] [2026-11-19T09:41:23.144Z] [2026-11-19T09:41:23.144Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T09:41:23.144Z] ] *** [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (352 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (352 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (351 retries left). [2026-11-19T09:43:46.917Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (351 retries left). [2026-11-19T09:43:46.917Z] ok: [an-a01n02] [2026-11-19T09:43:46.917Z] ok: [an-a01n01] [2026-11-19T09:43:46.917Z] [2026-11-19T09:43:46.917Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T09:43:46.917Z] ] *** [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-11-19T09:46:10.759Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-11-19T09:46:10.759Z] ok: [an-a01n02] [2026-11-19T09:46:10.759Z] ok: [an-a01n01] [2026-11-19T09:46:10.759Z] [2026-11-19T09:46:10.759Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T09:46:10.759Z] ] *** [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-11-19T09:48:51.140Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-11-19T09:48:51.140Z] ok: [an-a01n02] [2026-11-19T09:48:51.140Z] ok: [an-a01n01] [2026-11-19T09:48:51.140Z] [2026-11-19T09:48:51.140Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-11-19T09:48:51.140Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:48:51.140Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:48:51.140Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:48:51.140Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:48:51.140Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:48:51.140Z] [2026-11-19T09:48:51.140Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T09:48:51.140Z] ] *** [2026-11-19T09:48:51.875Z] ok: [an-a01n01] [2026-11-19T09:48:51.875Z] ok: [an-a01n02] [2026-11-19T09:48:51.875Z] [2026-11-19T09:48:51.875Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T09:48:51.875Z] ] *** [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (352 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (352 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (351 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (351 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (350 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (350 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (349 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (349 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (348 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (348 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (347 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (347 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (346 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (346 retries left). [2026-11-19T09:52:00.889Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (345 retries left). [2026-11-19T09:52:00.889Z] ok: [an-a01n01] [2026-11-19T09:52:00.889Z] ok: [an-a01n02] [2026-11-19T09:52:00.889Z] [2026-11-19T09:52:00.889Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T09:52:00.889Z] ] *** [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-11-19T09:53:54.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-11-19T09:53:54.467Z] ok: [an-a01n01] [2026-11-19T09:53:54.467Z] ok: [an-a01n02] [2026-11-19T09:53:54.467Z] [2026-11-19T09:53:54.467Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T09:53:54.467Z] ] *** [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-11-19T09:55:48.078Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-11-19T09:55:48.078Z] ok: [an-a01n01] [2026-11-19T09:55:48.078Z] ok: [an-a01n02] [2026-11-19T09:55:48.078Z] [2026-11-19T09:55:48.078Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T09:55:48.078Z] ] *** [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-11-19T09:57:41.622Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-11-19T09:57:41.622Z] ok: [an-a01n02] [2026-11-19T09:57:41.622Z] ok: [an-a01n01] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-11-19T09:57:41.622Z] ok: [an-a01n01] [2026-11-19T09:57:41.622Z] ok: [an-a01n02] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T09:57:41.622Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T09:57:41.622Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T09:57:41.622Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T09:57:41.622Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T09:57:41.622Z] echo "pcs:" [2026-11-19T09:57:41.622Z] echo "$pcsstatus" [2026-11-19T09:57:41.622Z] exit 1 [2026-11-19T09:57:41.622Z] fi [2026-11-19T09:57:41.622Z] exit 0 [2026-11-19T09:57:41.622Z] ] *** [2026-11-19T09:57:41.622Z] ok: [an-a01n02] [2026-11-19T09:57:41.622Z] ok: [an-a01n01] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T09:57:41.622Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T09:57:41.622Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T09:57:41.622Z] ret=0 [2026-11-19T09:57:41.622Z] if [ "$servers" != "1" ]; then [2026-11-19T09:57:41.622Z] echo "Not all servers are running on the same node!" [2026-11-19T09:57:41.622Z] echo "$pcsstatus" [2026-11-19T09:57:41.622Z] ret=1 [2026-11-19T09:57:41.622Z] fi [2026-11-19T09:57:41.622Z] echo "$hypervisor" [2026-11-19T09:57:41.622Z] exit $ret [2026-11-19T09:57:41.622Z] ] *** [2026-11-19T09:57:41.622Z] ok: [an-a01n02] [2026-11-19T09:57:41.622Z] ok: [an-a01n01] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-11-19T09:57:41.622Z] ok: [an-a01n01] [2026-11-19T09:57:41.622Z] ok: [an-a01n02] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T09:57:41.622Z] ] ********** [2026-11-19T09:57:41.622Z] skipping: [an-a01n02] [2026-11-19T09:57:41.622Z] changed: [an-a01n01] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T09:57:41.622Z] skipping: [an-a01n01] [2026-11-19T09:57:41.622Z] skipping: [an-a01n02] [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-11-19T09:57:41.622Z] skipping: [an-a01n01] [2026-11-19T09:57:41.622Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-11-19T09:57:41.622Z] [2026-11-19T09:57:41.622Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-11-19T09:57:41.622Z] ] *** [2026-11-19T09:58:51.197Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-11-19T09:58:51.197Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-11-19T09:58:51.197Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-11-19T09:58:51.197Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-11-19T09:58:51.197Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-11-19T09:58:51.197Z] ok: [an-a01n02] [2026-11-19T09:58:51.197Z] [2026-11-19T09:58:51.197Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T09:58:51.197Z] skipping: [an-a01n01] [2026-11-19T09:58:51.197Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-11-19T09:58:51.197Z] [2026-11-19T09:58:51.197Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T09:58:51.197Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T09:58:51.197Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T09:58:51.197Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T09:58:51.197Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T09:58:51.197Z] echo "pcs:" [2026-11-19T09:58:51.197Z] echo "$pcsstatus" [2026-11-19T09:58:51.197Z] exit 1 [2026-11-19T09:58:51.197Z] fi [2026-11-19T09:58:51.197Z] exit 0 [2026-11-19T09:58:51.197Z] ] *** [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-11-19T10:00:29.667Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:00:29.667Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:00:29.667Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:00:29.667Z] ret=0 [2026-11-19T10:00:29.667Z] if [ "$servers" != "1" ]; then [2026-11-19T10:00:29.667Z] echo "Not all servers are running on the same node!" [2026-11-19T10:00:29.667Z] echo "$pcsstatus" [2026-11-19T10:00:29.667Z] ret=1 [2026-11-19T10:00:29.667Z] fi [2026-11-19T10:00:29.667Z] echo "$hypervisor" [2026-11-19T10:00:29.667Z] exit $ret [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] => (item=1) [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] => (item=2) [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] => (item=3) [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] => (item=4) [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] => (item=5) [2026-11-19T10:00:29.667Z] skipping: [an-a01n01] [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:00:29.667Z] ] *** [2026-11-19T10:00:29.667Z] ok: [an-a01n02] [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-11-19T10:00:29.667Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:00:29.667Z] [2026-11-19T10:00:29.667Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T10:03:38.708Z] ok: [an-a01n02 -> localhost] [2026-11-19T10:03:38.708Z] ok: [an-a01n01 -> localhost] [2026-11-19T10:03:38.708Z] [2026-11-19T10:03:38.708Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-11-19T10:03:38.708Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:03:38.708Z] [2026-11-19T10:03:38.708Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:03:38.708Z] ] *** [2026-11-19T10:03:43.572Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:03:43.572Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:03:43.572Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:03:43.572Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:03:43.572Z] [2026-11-19T10:03:43.572Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:03:43.572Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T10:03:43.572Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:03:43.572Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:03:43.572Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:03:43.572Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:03:43.572Z] [2026-11-19T10:03:43.572Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:03:43.572Z] ] *** [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-11-19T10:07:08.720Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-11-19T10:07:08.720Z] ok: [an-a01n02] [2026-11-19T10:07:08.720Z] ok: [an-a01n01] [2026-11-19T10:07:08.720Z] [2026-11-19T10:07:08.720Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:07:08.720Z] ] *** [2026-11-19T10:07:19.707Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-11-19T10:07:19.707Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-11-19T10:07:19.707Z] ok: [an-a01n01] [2026-11-19T10:07:19.707Z] ok: [an-a01n02] [2026-11-19T10:07:19.707Z] [2026-11-19T10:07:19.707Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:07:19.707Z] ] *** [2026-11-19T10:07:19.707Z] ok: [an-a01n01] [2026-11-19T10:07:19.707Z] ok: [an-a01n02] [2026-11-19T10:07:19.707Z] [2026-11-19T10:07:19.707Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:07:19.707Z] ] *** [2026-11-19T10:07:19.707Z] ok: [an-a01n02] [2026-11-19T10:07:19.707Z] ok: [an-a01n01] [2026-11-19T10:07:19.707Z] [2026-11-19T10:07:19.707Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:07:19.707Z] ] *** [2026-11-19T10:07:20.442Z] ok: [an-a01n01] [2026-11-19T10:07:20.442Z] ok: [an-a01n02] [2026-11-19T10:07:20.442Z] [2026-11-19T10:07:20.442Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-11-19T10:07:20.442Z] ok: [an-a01n01] [2026-11-19T10:07:20.443Z] ok: [an-a01n02] [2026-11-19T10:07:20.443Z] [2026-11-19T10:07:20.443Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:07:20.785Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:07:20.785Z] [2026-11-19T10:07:20.785Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:07:20.785Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:07:20.785Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:07:20.785Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:07:20.785Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:07:20.785Z] echo "pcs:" [2026-11-19T10:07:20.785Z] echo "$pcsstatus" [2026-11-19T10:07:20.785Z] exit 1 [2026-11-19T10:07:20.785Z] fi [2026-11-19T10:07:20.785Z] exit 0 [2026-11-19T10:07:20.786Z] ] *** [2026-11-19T10:07:22.499Z] ok: [an-a01n01] [2026-11-19T10:07:22.499Z] ok: [an-a01n02] [2026-11-19T10:07:22.499Z] [2026-11-19T10:07:22.499Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:07:22.499Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:07:22.499Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:07:22.499Z] ret=0 [2026-11-19T10:07:22.499Z] if [ "$servers" != "1" ]; then [2026-11-19T10:07:22.499Z] echo "Not all servers are running on the same node!" [2026-11-19T10:07:22.499Z] echo "$pcsstatus" [2026-11-19T10:07:22.499Z] ret=1 [2026-11-19T10:07:22.499Z] fi [2026-11-19T10:07:22.499Z] echo "$hypervisor" [2026-11-19T10:07:22.499Z] exit $ret [2026-11-19T10:07:22.499Z] ] *** [2026-11-19T10:07:24.822Z] ok: [an-a01n01] [2026-11-19T10:07:24.822Z] ok: [an-a01n02] [2026-11-19T10:07:24.822Z] [2026-11-19T10:07:24.822Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-11-19T10:07:24.822Z] ok: [an-a01n01] [2026-11-19T10:07:24.822Z] ok: [an-a01n02] [2026-11-19T10:07:24.822Z] [2026-11-19T10:07:24.822Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-11-19T10:07:24.822Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:07:24.822Z] [2026-11-19T10:07:24.822Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T10:07:24.822Z] ] ********** [2026-11-19T10:07:25.556Z] skipping: [an-a01n01] [2026-11-19T10:07:25.556Z] changed: [an-a01n02] [2026-11-19T10:07:25.556Z] [2026-11-19T10:07:25.556Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T10:07:25.556Z] skipping: [an-a01n01] [2026-11-19T10:07:25.556Z] skipping: [an-a01n02] [2026-11-19T10:07:25.556Z] [2026-11-19T10:07:25.556Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-11-19T10:07:25.556Z] skipping: [an-a01n02] [2026-11-19T10:07:25.556Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-11-19T10:07:25.556Z] [2026-11-19T10:07:25.556Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-11-19T10:07:25.556Z] ] *** [2026-11-19T10:08:35.119Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-11-19T10:08:35.119Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-11-19T10:08:35.119Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-11-19T10:08:35.119Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-11-19T10:08:35.119Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-11-19T10:08:35.119Z] ok: [an-a01n01] [2026-11-19T10:08:35.119Z] [2026-11-19T10:08:35.119Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:08:35.119Z] skipping: [an-a01n02] [2026-11-19T10:08:35.119Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-11-19T10:08:35.119Z] [2026-11-19T10:08:35.119Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:08:35.119Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:08:35.119Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:08:35.119Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:08:35.119Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:08:35.119Z] echo "pcs:" [2026-11-19T10:08:35.119Z] echo "$pcsstatus" [2026-11-19T10:08:35.119Z] exit 1 [2026-11-19T10:08:35.119Z] fi [2026-11-19T10:08:35.119Z] exit 0 [2026-11-19T10:08:35.119Z] ] *** [2026-11-19T10:09:33.118Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-11-19T10:09:33.118Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-11-19T10:09:33.118Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-11-19T10:09:33.118Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-11-19T10:09:33.118Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-11-19T10:09:33.118Z] ok: [an-a01n01] [2026-11-19T10:09:33.118Z] [2026-11-19T10:09:33.118Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:09:33.118Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:09:33.118Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:09:33.118Z] ret=0 [2026-11-19T10:09:33.118Z] if [ "$servers" != "1" ]; then [2026-11-19T10:09:33.118Z] echo "Not all servers are running on the same node!" [2026-11-19T10:09:33.118Z] echo "$pcsstatus" [2026-11-19T10:09:33.118Z] ret=1 [2026-11-19T10:09:33.118Z] fi [2026-11-19T10:09:33.118Z] echo "$hypervisor" [2026-11-19T10:09:33.118Z] exit $ret [2026-11-19T10:09:33.118Z] ] *** [2026-11-19T10:09:33.118Z] ok: [an-a01n01] [2026-11-19T10:09:33.118Z] [2026-11-19T10:09:33.118Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] => (item=1) [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] => (item=2) [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] => (item=3) [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] => (item=4) [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] => (item=5) [2026-11-19T10:09:33.118Z] skipping: [an-a01n02] [2026-11-19T10:09:33.118Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-11-19T10:09:33.118Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-11-19T10:09:33.118Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-11-19T10:09:33.118Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-11-19T10:09:33.118Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-11-19T10:09:33.118Z] [2026-11-19T10:09:33.118Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:09:33.118Z] ] *** [2026-11-19T10:09:33.853Z] ok: [an-a01n01] [2026-11-19T10:09:33.853Z] [2026-11-19T10:09:33.853Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:09:33.853Z] ] *** [2026-11-19T10:09:36.427Z] ok: [an-a01n01] [2026-11-19T10:09:36.427Z] [2026-11-19T10:09:36.427Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:09:36.427Z] ] *** [2026-11-19T10:09:39.467Z] ok: [an-a01n01] [2026-11-19T10:09:39.467Z] [2026-11-19T10:09:39.467Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:09:39.467Z] ] *** [2026-11-19T10:09:40.202Z] ok: [an-a01n01] [2026-11-19T10:09:40.202Z] [2026-11-19T10:09:40.202Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:09:40.202Z] ] *** [2026-11-19T10:09:40.937Z] ok: [an-a01n01] [2026-11-19T10:09:40.937Z] [2026-11-19T10:09:40.937Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-11-19T10:09:40.937Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:09:40.937Z] [2026-11-19T10:09:40.937Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T10:13:05.054Z] ok: [an-a01n02 -> localhost] [2026-11-19T10:13:05.054Z] ok: [an-a01n01 -> localhost] [2026-11-19T10:13:05.054Z] [2026-11-19T10:13:05.054Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:13:05.054Z] [2026-11-19T10:13:05.054Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:13:05.054Z] ] *** [2026-11-19T10:13:05.054Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:13:05.054Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:13:05.054Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:13:05.054Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:13:05.054Z] [2026-11-19T10:13:05.054Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:13:05.054Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:13:05.054Z] [2026-11-19T10:13:05.055Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:13:05.055Z] ] *** [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (337 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (337 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (336 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (336 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (335 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (335 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (334 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (334 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (333 retries left). [2026-11-19T10:17:59.749Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (333 retries left). [2026-11-19T10:17:59.749Z] ok: [an-a01n01] [2026-11-19T10:17:59.749Z] ok: [an-a01n02] [2026-11-19T10:17:59.749Z] [2026-11-19T10:17:59.749Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:17:59.749Z] ] *** [2026-11-19T10:18:10.756Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-11-19T10:18:10.756Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-11-19T10:18:10.756Z] ok: [an-a01n02] [2026-11-19T10:18:10.756Z] ok: [an-a01n01] [2026-11-19T10:18:10.756Z] [2026-11-19T10:18:10.756Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:18:10.756Z] ] *** [2026-11-19T10:18:10.756Z] ok: [an-a01n01] [2026-11-19T10:18:10.756Z] ok: [an-a01n02] [2026-11-19T10:18:10.756Z] [2026-11-19T10:18:10.756Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:18:10.756Z] ] *** [2026-11-19T10:18:10.756Z] ok: [an-a01n01] [2026-11-19T10:18:10.756Z] ok: [an-a01n02] [2026-11-19T10:18:10.756Z] [2026-11-19T10:18:10.756Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:18:10.756Z] ] *** [2026-11-19T10:18:11.492Z] ok: [an-a01n01] [2026-11-19T10:18:11.492Z] ok: [an-a01n02] [2026-11-19T10:18:11.492Z] [2026-11-19T10:18:11.492Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-11-19T10:18:11.836Z] ok: [an-a01n01] [2026-11-19T10:18:11.836Z] ok: [an-a01n02] [2026-11-19T10:18:11.836Z] [2026-11-19T10:18:11.836Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:18:11.836Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:18:11.836Z] [2026-11-19T10:18:11.836Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:18:11.836Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:18:11.836Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:18:11.836Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:18:11.836Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:18:11.836Z] echo "pcs:" [2026-11-19T10:18:11.836Z] echo "$pcsstatus" [2026-11-19T10:18:11.836Z] exit 1 [2026-11-19T10:18:11.836Z] fi [2026-11-19T10:18:11.836Z] exit 0 [2026-11-19T10:18:11.836Z] ] *** [2026-11-19T10:18:14.159Z] ok: [an-a01n01] [2026-11-19T10:18:14.159Z] ok: [an-a01n02] [2026-11-19T10:18:14.159Z] [2026-11-19T10:18:14.159Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:18:14.159Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:18:14.159Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:18:14.159Z] ret=0 [2026-11-19T10:18:14.159Z] if [ "$servers" != "1" ]; then [2026-11-19T10:18:14.159Z] echo "Not all servers are running on the same node!" [2026-11-19T10:18:14.159Z] echo "$pcsstatus" [2026-11-19T10:18:14.159Z] ret=1 [2026-11-19T10:18:14.159Z] fi [2026-11-19T10:18:14.159Z] echo "$hypervisor" [2026-11-19T10:18:14.159Z] exit $ret [2026-11-19T10:18:14.159Z] ] *** [2026-11-19T10:18:15.870Z] ok: [an-a01n02] [2026-11-19T10:18:15.870Z] ok: [an-a01n01] [2026-11-19T10:18:15.870Z] [2026-11-19T10:18:15.870Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-11-19T10:18:15.870Z] ok: [an-a01n01] [2026-11-19T10:18:15.870Z] ok: [an-a01n02] [2026-11-19T10:18:15.870Z] [2026-11-19T10:18:15.870Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-11-19T10:18:15.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:18:15.870Z] [2026-11-19T10:18:15.870Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T10:18:15.870Z] ] ********** [2026-11-19T10:18:16.605Z] skipping: [an-a01n02] [2026-11-19T10:18:16.605Z] changed: [an-a01n01] [2026-11-19T10:18:16.605Z] [2026-11-19T10:18:16.605Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T10:18:16.605Z] skipping: [an-a01n01] [2026-11-19T10:18:16.605Z] skipping: [an-a01n02] [2026-11-19T10:18:16.605Z] [2026-11-19T10:18:16.605Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-11-19T10:18:16.947Z] skipping: [an-a01n01] [2026-11-19T10:18:16.947Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-11-19T10:18:16.947Z] [2026-11-19T10:18:16.947Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-11-19T10:18:16.947Z] ] *** [2026-11-19T10:19:26.483Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-11-19T10:19:26.483Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-11-19T10:19:26.483Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-11-19T10:19:26.483Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-11-19T10:19:26.483Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-11-19T10:19:26.483Z] ok: [an-a01n02] [2026-11-19T10:19:26.483Z] [2026-11-19T10:19:26.483Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:19:26.483Z] skipping: [an-a01n01] [2026-11-19T10:19:26.483Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-11-19T10:19:26.483Z] [2026-11-19T10:19:26.483Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:19:26.483Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:19:26.483Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:19:26.483Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:19:26.483Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:19:26.483Z] echo "pcs:" [2026-11-19T10:19:26.483Z] echo "$pcsstatus" [2026-11-19T10:19:26.483Z] exit 1 [2026-11-19T10:19:26.483Z] fi [2026-11-19T10:19:26.483Z] exit 0 [2026-11-19T10:19:26.483Z] ] *** [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-11-19T10:20:36.025Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-11-19T10:20:36.025Z] ok: [an-a01n02] [2026-11-19T10:20:36.025Z] [2026-11-19T10:20:36.025Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:20:36.025Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:20:36.025Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:20:36.025Z] ret=0 [2026-11-19T10:20:36.025Z] if [ "$servers" != "1" ]; then [2026-11-19T10:20:36.025Z] echo "Not all servers are running on the same node!" [2026-11-19T10:20:36.025Z] echo "$pcsstatus" [2026-11-19T10:20:36.025Z] ret=1 [2026-11-19T10:20:36.025Z] fi [2026-11-19T10:20:36.025Z] echo "$hypervisor" [2026-11-19T10:20:36.025Z] exit $ret [2026-11-19T10:20:36.025Z] ] *** [2026-11-19T10:20:36.025Z] ok: [an-a01n02] [2026-11-19T10:20:36.025Z] [2026-11-19T10:20:36.025Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] => (item=1) [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] => (item=2) [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] => (item=3) [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] => (item=4) [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] => (item=5) [2026-11-19T10:20:36.025Z] skipping: [an-a01n01] [2026-11-19T10:20:36.025Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-11-19T10:20:36.025Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-11-19T10:20:36.025Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-11-19T10:20:36.025Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-11-19T10:20:36.025Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-11-19T10:20:36.025Z] [2026-11-19T10:20:36.025Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:20:36.025Z] ] *** [2026-11-19T10:20:36.760Z] ok: [an-a01n02] [2026-11-19T10:20:36.760Z] [2026-11-19T10:20:36.760Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:20:36.760Z] ] *** [2026-11-19T10:20:37.495Z] ok: [an-a01n02] [2026-11-19T10:20:37.495Z] [2026-11-19T10:20:37.495Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:20:37.495Z] ] *** [2026-11-19T10:20:38.229Z] ok: [an-a01n02] [2026-11-19T10:20:38.229Z] [2026-11-19T10:20:38.229Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:20:38.229Z] ] *** [2026-11-19T10:20:39.416Z] ok: [an-a01n02] [2026-11-19T10:20:39.416Z] [2026-11-19T10:20:39.416Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:20:39.416Z] ] *** [2026-11-19T10:20:40.150Z] ok: [an-a01n02] [2026-11-19T10:20:40.150Z] [2026-11-19T10:20:40.150Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-11-19T10:20:40.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:20:40.150Z] [2026-11-19T10:20:40.150Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01 -> localhost] [2026-11-19T10:24:04.774Z] ok: [an-a01n02 -> localhost] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:24:04.774Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:24:04.774Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:24:04.774Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:24:04.774Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:24:04.774Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:24:04.774Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:24:04.774Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:24:04.774Z] echo "pcs:" [2026-11-19T10:24:04.774Z] echo "$pcsstatus" [2026-11-19T10:24:04.774Z] exit 1 [2026-11-19T10:24:04.774Z] fi [2026-11-19T10:24:04.774Z] exit 0 [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:24:04.774Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:24:04.774Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:24:04.774Z] ret=0 [2026-11-19T10:24:04.774Z] if [ "$servers" != "1" ]; then [2026-11-19T10:24:04.774Z] echo "Not all servers are running on the same node!" [2026-11-19T10:24:04.774Z] echo "$pcsstatus" [2026-11-19T10:24:04.774Z] ret=1 [2026-11-19T10:24:04.774Z] fi [2026-11-19T10:24:04.774Z] echo "$hypervisor" [2026-11-19T10:24:04.774Z] exit $ret [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-11-19T10:24:04.774Z] ok: [an-a01n01] [2026-11-19T10:24:04.774Z] ok: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T10:24:04.774Z] ] ********** [2026-11-19T10:24:04.774Z] skipping: [an-a01n01] [2026-11-19T10:24:04.774Z] changed: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T10:24:04.774Z] skipping: [an-a01n01] [2026-11-19T10:24:04.774Z] skipping: [an-a01n02] [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-11-19T10:24:04.774Z] skipping: [an-a01n02] [2026-11-19T10:24:04.774Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-11-19T10:24:04.774Z] [2026-11-19T10:24:04.774Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-11-19T10:24:04.774Z] ] *** [2026-11-19T10:25:14.298Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-11-19T10:25:14.298Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-11-19T10:25:14.298Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-11-19T10:25:14.298Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-11-19T10:25:14.298Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-11-19T10:25:14.298Z] ok: [an-a01n01] [2026-11-19T10:25:14.298Z] [2026-11-19T10:25:14.298Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:25:14.298Z] skipping: [an-a01n02] [2026-11-19T10:25:14.298Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-11-19T10:25:14.298Z] [2026-11-19T10:25:14.298Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:25:14.298Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:25:14.298Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:25:14.298Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:25:14.298Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:25:14.298Z] echo "pcs:" [2026-11-19T10:25:14.298Z] echo "$pcsstatus" [2026-11-19T10:25:14.298Z] exit 1 [2026-11-19T10:25:14.298Z] fi [2026-11-19T10:25:14.298Z] exit 0 [2026-11-19T10:25:14.298Z] ] *** [2026-11-19T10:26:12.283Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-11-19T10:26:12.283Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-11-19T10:26:12.284Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-11-19T10:26:12.284Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-11-19T10:26:12.284Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-11-19T10:26:12.284Z] ok: [an-a01n01] [2026-11-19T10:26:12.284Z] [2026-11-19T10:26:12.284Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:26:12.284Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:26:12.284Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:26:12.284Z] ret=0 [2026-11-19T10:26:12.284Z] if [ "$servers" != "1" ]; then [2026-11-19T10:26:12.284Z] echo "Not all servers are running on the same node!" [2026-11-19T10:26:12.284Z] echo "$pcsstatus" [2026-11-19T10:26:12.284Z] ret=1 [2026-11-19T10:26:12.284Z] fi [2026-11-19T10:26:12.284Z] echo "$hypervisor" [2026-11-19T10:26:12.284Z] exit $ret [2026-11-19T10:26:12.284Z] ] *** [2026-11-19T10:26:12.284Z] ok: [an-a01n01] [2026-11-19T10:26:12.284Z] [2026-11-19T10:26:12.284Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] => (item=1) [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] => (item=2) [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] => (item=3) [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] => (item=4) [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] => (item=5) [2026-11-19T10:26:12.284Z] skipping: [an-a01n02] [2026-11-19T10:26:12.284Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-11-19T10:26:12.284Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-11-19T10:26:12.284Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-11-19T10:26:12.284Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-11-19T10:26:12.284Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-11-19T10:26:12.284Z] [2026-11-19T10:26:12.284Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:26:12.284Z] ] *** [2026-11-19T10:26:25.576Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-11-19T10:26:25.576Z] ok: [an-a01n01] [2026-11-19T10:26:25.576Z] [2026-11-19T10:26:25.576Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:26:25.576Z] ] *** [2026-11-19T10:26:25.576Z] ok: [an-a01n01] [2026-11-19T10:26:25.576Z] [2026-11-19T10:26:25.576Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:26:25.576Z] ] *** [2026-11-19T10:26:26.312Z] ok: [an-a01n01] [2026-11-19T10:26:26.312Z] [2026-11-19T10:26:26.312Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:26:26.312Z] ] *** [2026-11-19T10:26:27.046Z] ok: [an-a01n01] [2026-11-19T10:26:27.046Z] [2026-11-19T10:26:27.046Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:26:27.046Z] ] *** [2026-11-19T10:26:27.779Z] ok: [an-a01n01] [2026-11-19T10:26:27.779Z] [2026-11-19T10:26:27.779Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-11-19T10:26:27.779Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:26:27.779Z] [2026-11-19T10:26:27.780Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n01 -> localhost] [2026-11-19T10:29:51.889Z] ok: [an-a01n02 -> localhost] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:29:51.889Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:29:51.889Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:29:51.889Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:29:51.889Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:29:51.889Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:29:51.889Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:29:51.889Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:29:51.889Z] echo "pcs:" [2026-11-19T10:29:51.889Z] echo "$pcsstatus" [2026-11-19T10:29:51.889Z] exit 1 [2026-11-19T10:29:51.889Z] fi [2026-11-19T10:29:51.889Z] exit 0 [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:29:51.889Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:29:51.889Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:29:51.889Z] ret=0 [2026-11-19T10:29:51.889Z] if [ "$servers" != "1" ]; then [2026-11-19T10:29:51.889Z] echo "Not all servers are running on the same node!" [2026-11-19T10:29:51.889Z] echo "$pcsstatus" [2026-11-19T10:29:51.889Z] ret=1 [2026-11-19T10:29:51.889Z] fi [2026-11-19T10:29:51.889Z] echo "$hypervisor" [2026-11-19T10:29:51.889Z] exit $ret [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-11-19T10:29:51.889Z] ok: [an-a01n01] [2026-11-19T10:29:51.889Z] ok: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-11-19T10:29:51.889Z] ] ********** [2026-11-19T10:29:51.889Z] skipping: [an-a01n02] [2026-11-19T10:29:51.889Z] changed: [an-a01n01] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-11-19T10:29:51.889Z] skipping: [an-a01n01] [2026-11-19T10:29:51.889Z] skipping: [an-a01n02] [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-11-19T10:29:51.889Z] skipping: [an-a01n01] [2026-11-19T10:29:51.889Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-11-19T10:29:51.889Z] [2026-11-19T10:29:51.889Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-11-19T10:29:51.889Z] ] *** [2026-11-19T10:31:01.411Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-11-19T10:31:01.411Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-11-19T10:31:01.411Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-11-19T10:31:01.411Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-11-19T10:31:01.411Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-11-19T10:31:01.411Z] ok: [an-a01n02] [2026-11-19T10:31:01.411Z] [2026-11-19T10:31:01.411Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-11-19T10:31:01.411Z] skipping: [an-a01n01] [2026-11-19T10:31:01.411Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-11-19T10:31:01.411Z] [2026-11-19T10:31:01.411Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:31:01.411Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:31:01.411Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:31:01.411Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:31:01.411Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:31:01.411Z] echo "pcs:" [2026-11-19T10:31:01.411Z] echo "$pcsstatus" [2026-11-19T10:31:01.411Z] exit 1 [2026-11-19T10:31:01.411Z] fi [2026-11-19T10:31:01.411Z] exit 0 [2026-11-19T10:31:01.411Z] ] *** [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-11-19T10:32:10.933Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-11-19T10:32:10.933Z] ok: [an-a01n02] [2026-11-19T10:32:10.933Z] [2026-11-19T10:32:10.933Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:32:10.933Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:32:10.933Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:32:10.933Z] ret=0 [2026-11-19T10:32:10.933Z] if [ "$servers" != "1" ]; then [2026-11-19T10:32:10.933Z] echo "Not all servers are running on the same node!" [2026-11-19T10:32:10.933Z] echo "$pcsstatus" [2026-11-19T10:32:10.933Z] ret=1 [2026-11-19T10:32:10.933Z] fi [2026-11-19T10:32:10.933Z] echo "$hypervisor" [2026-11-19T10:32:10.933Z] exit $ret [2026-11-19T10:32:10.933Z] ] *** [2026-11-19T10:32:10.933Z] ok: [an-a01n02] [2026-11-19T10:32:10.933Z] [2026-11-19T10:32:10.933Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] => (item=1) [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] => (item=2) [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] => (item=3) [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] => (item=4) [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] => (item=5) [2026-11-19T10:32:10.933Z] skipping: [an-a01n01] [2026-11-19T10:32:10.933Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-11-19T10:32:10.933Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-11-19T10:32:10.933Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-11-19T10:32:10.933Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-11-19T10:32:10.933Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-11-19T10:32:10.933Z] [2026-11-19T10:32:10.933Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:32:10.933Z] ] *** [2026-11-19T10:32:24.590Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-11-19T10:32:24.590Z] ok: [an-a01n02] [2026-11-19T10:32:24.590Z] [2026-11-19T10:32:24.590Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:32:24.590Z] ] *** [2026-11-19T10:32:24.590Z] ok: [an-a01n02] [2026-11-19T10:32:24.590Z] [2026-11-19T10:32:24.590Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:32:24.590Z] ] *** [2026-11-19T10:32:24.931Z] ok: [an-a01n02] [2026-11-19T10:32:24.931Z] [2026-11-19T10:32:24.931Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:32:24.931Z] ] *** [2026-11-19T10:32:25.664Z] ok: [an-a01n02] [2026-11-19T10:32:25.664Z] [2026-11-19T10:32:25.664Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T10:32:25.664Z] ] *** [2026-11-19T10:32:26.397Z] ok: [an-a01n02] [2026-11-19T10:32:26.397Z] [2026-11-19T10:32:26.397Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-11-19T10:32:26.397Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-11-19T10:32:26.397Z] [2026-11-19T10:32:26.397Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01 -> localhost] [2026-11-19T10:35:50.593Z] ok: [an-a01n02 -> localhost] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:35:50.593Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:35:50.593Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:35:50.593Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] PLAY [basic server tests] ****************************************************** [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] [2026-11-19T10:35:50.593Z] ok: [an-a01n02] [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:50.593Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:35:50.593Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:35:50.593Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:35:50.593Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-11-19T10:35:50.593Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-11-19T10:35:50.593Z] [2026-11-19T10:35:50.593Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-11-19T10:35:50.593Z] if [ -z "$storage" ]; then [2026-11-19T10:35:50.593Z] echo "Unable to detect shared storage" [2026-11-19T10:35:50.593Z] exit 1 [2026-11-19T10:35:50.593Z] fi [2026-11-19T10:35:50.593Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:35:50.593Z] ] *** [2026-11-19T10:35:56.609Z] changed: [an-a01n01] [2026-11-19T10:35:56.609Z] changed: [an-a01n02] [2026-11-19T10:35:56.609Z] [2026-11-19T10:35:56.609Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-11-19T10:35:56.609Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-11-19T10:35:56.609Z] [2026-11-19T10:35:56.609Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-11-19T10:35:56.609Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-11-19T10:35:56.609Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-11-19T10:35:56.609Z] echo "Unable to detect all deploy iso" [2026-11-19T10:35:56.609Z] echo "Detected isos" [2026-11-19T10:35:56.609Z] echo "$isos" [2026-11-19T10:35:56.609Z] exit 1 [2026-11-19T10:35:56.610Z] fi [2026-11-19T10:35:56.610Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-11-19T10:35:56.610Z] if [ -z "$installiso" ]; then [2026-11-19T10:35:56.610Z] echo "Unable to detect install iso" [2026-11-19T10:35:56.610Z] echo "Detected isos" [2026-11-19T10:35:56.610Z] echo "$isos" [2026-11-19T10:35:56.610Z] exit 1 [2026-11-19T10:35:56.610Z] fi [2026-11-19T10:35:56.610Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:35:56.610Z] ] *** [2026-11-19T10:36:01.463Z] changed: [an-a01n01] [2026-11-19T10:36:01.463Z] changed: [an-a01n02] [2026-11-19T10:36:01.463Z] [2026-11-19T10:36:01.463Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-11-19T10:36:01.463Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-11-19T10:36:01.463Z] [2026-11-19T10:36:01.463Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:01.463Z] ] *** [2026-11-19T10:36:02.648Z] skipping: [an-a01n02] [2026-11-19T10:36:02.648Z] ok: [an-a01n01] [2026-11-19T10:36:02.648Z] [2026-11-19T10:36:02.648Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T10:36:02.648Z] --ci-test \ [2026-11-19T10:36:02.648Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T10:36:02.648Z] --os almalinux9 \ [2026-11-19T10:36:02.648Z] --cpu 2 --ram 4G \ [2026-11-19T10:36:02.648Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T10:36:02.648Z] --install-media {{ installisouuid }} \ [2026-11-19T10:36:02.648Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T10:36:02.648Z] ] *** [2026-11-19T10:36:02.648Z] skipping: [an-a01n01] [2026-11-19T10:36:02.648Z] skipping: [an-a01n02] [2026-11-19T10:36:02.648Z] [2026-11-19T10:36:02.648Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T10:36:02.648Z] skipping: [an-a01n01] [2026-11-19T10:36:02.648Z] skipping: [an-a01n02] [2026-11-19T10:36:02.648Z] [2026-11-19T10:36:02.648Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:36:02.648Z] skipping: [an-a01n01] [2026-11-19T10:36:02.648Z] skipping: [an-a01n02] [2026-11-19T10:36:02.648Z] [2026-11-19T10:36:02.648Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T10:36:02.990Z] skipping: [an-a01n01] [2026-11-19T10:36:02.990Z] skipping: [an-a01n02] [2026-11-19T10:36:02.990Z] [2026-11-19T10:36:02.990Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T10:36:02.990Z] skipping: [an-a01n01] [2026-11-19T10:36:02.990Z] skipping: [an-a01n02] [2026-11-19T10:36:02.990Z] [2026-11-19T10:36:02.990Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-11-19T10:36:02.990Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-11-19T10:36:02.990Z] [2026-11-19T10:36:02.990Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:02.990Z] ] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] ok: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T10:36:04.809Z] --ci-test \ [2026-11-19T10:36:04.809Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T10:36:04.809Z] --os almalinux9 \ [2026-11-19T10:36:04.809Z] --cpu 2 --ram 4G \ [2026-11-19T10:36:04.809Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T10:36:04.809Z] --install-media {{ installisouuid }} \ [2026-11-19T10:36:04.809Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T10:36:04.809Z] ] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] skipping: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] skipping: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] skipping: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] skipping: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T10:36:04.809Z] skipping: [an-a01n01] [2026-11-19T10:36:04.809Z] skipping: [an-a01n02] [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-11-19T10:36:04.809Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:36:04.809Z] [2026-11-19T10:36:04.809Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:36:04.809Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:36:04.809Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:36:04.809Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:36:04.809Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:36:04.809Z] echo "pcs:" [2026-11-19T10:36:04.809Z] echo "$pcsstatus" [2026-11-19T10:36:04.809Z] exit 1 [2026-11-19T10:36:04.809Z] fi [2026-11-19T10:36:04.809Z] exit 0 [2026-11-19T10:36:04.809Z] ] *** [2026-11-19T10:36:07.128Z] ok: [an-a01n01] [2026-11-19T10:36:07.128Z] ok: [an-a01n02] [2026-11-19T10:36:07.128Z] [2026-11-19T10:36:07.128Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:36:07.128Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:36:07.128Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:36:07.128Z] ret=0 [2026-11-19T10:36:07.128Z] if [ "$servers" != "1" ]; then [2026-11-19T10:36:07.128Z] echo "Not all servers are running on the same node!" [2026-11-19T10:36:07.128Z] echo "$pcsstatus" [2026-11-19T10:36:07.128Z] ret=1 [2026-11-19T10:36:07.128Z] fi [2026-11-19T10:36:07.128Z] echo "$hypervisor" [2026-11-19T10:36:07.128Z] exit $ret [2026-11-19T10:36:07.128Z] ] *** [2026-11-19T10:36:08.837Z] ok: [an-a01n02] [2026-11-19T10:36:08.837Z] ok: [an-a01n01] [2026-11-19T10:36:08.837Z] [2026-11-19T10:36:08.837Z] PLAY [delete extra server tests] *********************************************** [2026-11-19T10:36:08.837Z] [2026-11-19T10:36:08.837Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:36:10.545Z] ok: [an-a01n01] [2026-11-19T10:36:10.545Z] ok: [an-a01n02] [2026-11-19T10:36:10.545Z] [2026-11-19T10:36:10.545Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T10:36:10.887Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:36:10.887Z] [2026-11-19T10:36:10.887Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:36:10.887Z] ] *** [2026-11-19T10:36:14.755Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:36:14.755Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:36:14.755Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:36:14.755Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:36:14.755Z] [2026-11-19T10:36:14.755Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-11-19T10:36:14.755Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T10:36:14.755Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T10:36:14.755Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T10:36:14.755Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T10:36:14.755Z] [2026-11-19T10:36:14.755Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:14.755Z] ] *** [2026-11-19T10:36:16.464Z] skipping: [an-a01n01] [2026-11-19T10:36:16.464Z] ok: [an-a01n02] [2026-11-19T10:36:16.464Z] [2026-11-19T10:36:16.464Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T10:36:16.464Z] --force \ [2026-11-19T10:36:16.464Z] --server an-test-deploy{{ servernum }} [2026-11-19T10:36:16.464Z] ] *** [2026-11-19T10:36:22.484Z] skipping: [an-a01n01] [2026-11-19T10:36:22.484Z] changed: [an-a01n02] [2026-11-19T10:36:22.484Z] [2026-11-19T10:36:22.484Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:22.484Z] ] *** [2026-11-19T10:36:50.442Z] skipping: [an-a01n01] [2026-11-19T10:36:50.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-11-19T10:36:50.442Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-11-19T10:36:50.442Z] ok: [an-a01n02] [2026-11-19T10:36:50.442Z] [2026-11-19T10:36:50.442Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T10:36:50.442Z] ] *** [2026-11-19T10:36:50.442Z] skipping: [an-a01n01] [2026-11-19T10:36:50.442Z] ok: [an-a01n02] [2026-11-19T10:36:50.442Z] [2026-11-19T10:36:50.442Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T10:36:50.442Z] ] *** [2026-11-19T10:36:50.442Z] skipping: [an-a01n01] [2026-11-19T10:36:50.442Z] ok: [an-a01n02] [2026-11-19T10:36:50.442Z] [2026-11-19T10:36:50.442Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:50.442Z] ] *** [2026-11-19T10:36:50.442Z] skipping: [an-a01n01] [2026-11-19T10:36:50.442Z] ok: [an-a01n02] [2026-11-19T10:36:50.442Z] [2026-11-19T10:36:50.442Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T10:36:50.442Z] --force \ [2026-11-19T10:36:50.442Z] --server an-test-deploy{{ servernum }} [2026-11-19T10:36:50.442Z] ] *** [2026-11-19T10:36:56.462Z] skipping: [an-a01n01] [2026-11-19T10:36:56.462Z] changed: [an-a01n02] [2026-11-19T10:36:56.462Z] [2026-11-19T10:36:56.462Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:36:56.462Z] ] *** [2026-11-19T10:38:19.813Z] skipping: [an-a01n01] [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-11-19T10:38:19.813Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-11-19T10:38:19.813Z] ok: [an-a01n02] [2026-11-19T10:38:19.813Z] [2026-11-19T10:38:19.813Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T10:38:19.813Z] ] *** [2026-11-19T10:38:19.813Z] skipping: [an-a01n01] [2026-11-19T10:38:19.813Z] ok: [an-a01n02] [2026-11-19T10:38:19.813Z] [2026-11-19T10:38:19.813Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T10:38:19.813Z] ] *** [2026-11-19T10:38:19.813Z] skipping: [an-a01n01] [2026-11-19T10:38:19.813Z] ok: [an-a01n02] [2026-11-19T10:38:19.813Z] [2026-11-19T10:38:19.813Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:38:19.813Z] ] *** [2026-11-19T10:38:19.813Z] skipping: [an-a01n01] [2026-11-19T10:38:19.813Z] ok: [an-a01n02] [2026-11-19T10:38:19.813Z] [2026-11-19T10:38:19.813Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T10:38:19.813Z] --force \ [2026-11-19T10:38:19.813Z] --server an-test-deploy{{ servernum }} [2026-11-19T10:38:19.813Z] ] *** [2026-11-19T10:38:19.813Z] skipping: [an-a01n01] [2026-11-19T10:38:19.813Z] changed: [an-a01n02] [2026-11-19T10:38:19.813Z] [2026-11-19T10:38:19.813Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:38:19.813Z] ] *** [2026-11-19T10:39:17.809Z] skipping: [an-a01n01] [2026-11-19T10:39:17.809Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-11-19T10:39:17.809Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-11-19T10:39:17.809Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-11-19T10:39:17.809Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-11-19T10:39:17.809Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-11-19T10:39:17.809Z] ok: [an-a01n02] [2026-11-19T10:39:17.809Z] [2026-11-19T10:39:17.809Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T10:39:17.809Z] ] *** [2026-11-19T10:39:17.809Z] skipping: [an-a01n01] [2026-11-19T10:39:17.809Z] ok: [an-a01n02] [2026-11-19T10:39:17.809Z] [2026-11-19T10:39:17.809Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T10:39:17.809Z] ] *** [2026-11-19T10:39:17.809Z] skipping: [an-a01n01] [2026-11-19T10:39:17.809Z] ok: [an-a01n02] [2026-11-19T10:39:17.809Z] [2026-11-19T10:39:17.809Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:39:17.809Z] ] *** [2026-11-19T10:39:17.809Z] skipping: [an-a01n01] [2026-11-19T10:39:17.809Z] ok: [an-a01n02] [2026-11-19T10:39:17.809Z] [2026-11-19T10:39:17.809Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T10:39:17.809Z] --force \ [2026-11-19T10:39:17.809Z] --server an-test-deploy{{ servernum }} [2026-11-19T10:39:17.809Z] ] *** [2026-11-19T10:39:23.920Z] skipping: [an-a01n01] [2026-11-19T10:39:23.920Z] changed: [an-a01n02] [2026-11-19T10:39:23.920Z] [2026-11-19T10:39:23.920Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:39:23.920Z] ] *** [2026-11-19T10:40:33.437Z] skipping: [an-a01n01] [2026-11-19T10:40:33.437Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-11-19T10:40:33.437Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-11-19T10:40:33.437Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-11-19T10:40:33.437Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-11-19T10:40:33.437Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-11-19T10:40:33.437Z] ok: [an-a01n02] [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T10:40:33.437Z] ] *** [2026-11-19T10:40:33.437Z] skipping: [an-a01n01] [2026-11-19T10:40:33.437Z] ok: [an-a01n02] [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T10:40:33.437Z] ] *** [2026-11-19T10:40:33.437Z] skipping: [an-a01n01] [2026-11-19T10:40:33.437Z] ok: [an-a01n02] [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] PLAY [create server for DR testing] ******************************************** [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:40:33.437Z] ok: [an-a01n02] [2026-11-19T10:40:33.437Z] ok: [an-a01n01] [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T10:40:33.437Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:40:33.437Z] ] *** [2026-11-19T10:40:33.437Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:40:33.437Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:40:33.437Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:40:33.437Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-11-19T10:40:33.437Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:33.437Z] [2026-11-19T10:40:33.437Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-11-19T10:40:33.437Z] if [ -z "$storage" ]; then [2026-11-19T10:40:33.437Z] echo "Unable to detect shared storage" [2026-11-19T10:40:33.437Z] exit 1 [2026-11-19T10:40:33.437Z] fi [2026-11-19T10:40:33.437Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:40:33.437Z] ] *** [2026-11-19T10:40:35.148Z] changed: [an-a01n02] [2026-11-19T10:40:35.148Z] changed: [an-a01n01] [2026-11-19T10:40:35.148Z] [2026-11-19T10:40:35.148Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-11-19T10:40:35.148Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:35.148Z] [2026-11-19T10:40:35.148Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-11-19T10:40:35.148Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-11-19T10:40:35.148Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-11-19T10:40:35.148Z] echo "Unable to detect all deploy iso" [2026-11-19T10:40:35.148Z] echo "Detected isos" [2026-11-19T10:40:35.148Z] echo "$isos" [2026-11-19T10:40:35.148Z] exit 1 [2026-11-19T10:40:35.148Z] fi [2026-11-19T10:40:35.148Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-11-19T10:40:35.148Z] if [ -z "$installiso" ]; then [2026-11-19T10:40:35.148Z] echo "Unable to detect install iso" [2026-11-19T10:40:35.148Z] echo "Detected isos" [2026-11-19T10:40:35.148Z] echo "$isos" [2026-11-19T10:40:35.148Z] exit 1 [2026-11-19T10:40:35.148Z] fi [2026-11-19T10:40:35.148Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:40:35.148Z] ] *** [2026-11-19T10:40:40.094Z] changed: [an-a01n02] [2026-11-19T10:40:40.094Z] changed: [an-a01n01] [2026-11-19T10:40:40.094Z] [2026-11-19T10:40:40.094Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-11-19T10:40:40.435Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:40.435Z] [2026-11-19T10:40:40.435Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:40:40.435Z] ] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] ok: [an-a01n01] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T10:40:42.147Z] --ci-test \ [2026-11-19T10:40:42.147Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T10:40:42.147Z] --os almalinux9 \ [2026-11-19T10:40:42.147Z] --cpu 2 --ram 4G \ [2026-11-19T10:40:42.147Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T10:40:42.147Z] --install-media {{ installisouuid }} \ [2026-11-19T10:40:42.147Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T10:40:42.147Z] ] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n01] [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n01] [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n01] [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n01] [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T10:40:42.147Z] skipping: [an-a01n01] [2026-11-19T10:40:42.147Z] skipping: [an-a01n02] [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-11-19T10:40:42.147Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:42.147Z] [2026-11-19T10:40:42.147Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:40:42.147Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:40:42.147Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:40:42.147Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:40:42.147Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:40:42.147Z] echo "pcs:" [2026-11-19T10:40:42.147Z] echo "$pcsstatus" [2026-11-19T10:40:42.147Z] exit 1 [2026-11-19T10:40:42.147Z] fi [2026-11-19T10:40:42.147Z] exit 0 [2026-11-19T10:40:42.147Z] ] *** [2026-11-19T10:40:44.465Z] ok: [an-a01n01] [2026-11-19T10:40:44.465Z] ok: [an-a01n02] [2026-11-19T10:40:44.465Z] [2026-11-19T10:40:44.465Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:40:44.465Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:40:44.465Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:40:44.465Z] ret=0 [2026-11-19T10:40:44.465Z] if [ "$servers" != "1" ]; then [2026-11-19T10:40:44.465Z] echo "Not all servers are running on the same node!" [2026-11-19T10:40:44.466Z] echo "$pcsstatus" [2026-11-19T10:40:44.466Z] ret=1 [2026-11-19T10:40:44.466Z] fi [2026-11-19T10:40:44.466Z] echo "$hypervisor" [2026-11-19T10:40:44.466Z] exit $ret [2026-11-19T10:40:44.466Z] ] *** [2026-11-19T10:40:46.176Z] ok: [an-a01n01] [2026-11-19T10:40:46.176Z] ok: [an-a01n02] [2026-11-19T10:40:46.176Z] [2026-11-19T10:40:46.176Z] PLAY [DR tests] **************************************************************** [2026-11-19T10:40:46.176Z] skipping: no hosts matched [2026-11-19T10:40:46.176Z] [2026-11-19T10:40:46.176Z] PLAY [upgrade server test] ***************************************************** [2026-11-19T10:40:46.176Z] [2026-11-19T10:40:46.176Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:40:47.885Z] ok: [an-a01n01] [2026-11-19T10:40:47.885Z] ok: [an-a01n02] [2026-11-19T10:40:47.885Z] [2026-11-19T10:40:47.885Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T10:40:47.885Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:47.885Z] [2026-11-19T10:40:47.885Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T10:40:47.885Z] ] *** [2026-11-19T10:40:51.758Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T10:40:51.758Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T10:40:51.758Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T10:40:51.758Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T10:40:51.758Z] [2026-11-19T10:40:51.758Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-11-19T10:40:51.758Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:51.758Z] [2026-11-19T10:40:51.758Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-11-19T10:40:51.758Z] if [ -z "$storage" ]; then [2026-11-19T10:40:51.758Z] echo "Unable to detect shared storage" [2026-11-19T10:40:51.758Z] exit 1 [2026-11-19T10:40:51.758Z] fi [2026-11-19T10:40:51.758Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:40:51.758Z] ] *** [2026-11-19T10:40:57.868Z] changed: [an-a01n01] [2026-11-19T10:40:57.868Z] changed: [an-a01n02] [2026-11-19T10:40:57.868Z] [2026-11-19T10:40:57.868Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-11-19T10:40:57.868Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-11-19T10:40:57.868Z] [2026-11-19T10:40:57.868Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-11-19T10:40:57.868Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-11-19T10:40:57.868Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-11-19T10:40:57.868Z] echo "Unable to detect all deploy iso" [2026-11-19T10:40:57.868Z] echo "Detected isos" [2026-11-19T10:40:57.868Z] echo "$isos" [2026-11-19T10:40:57.868Z] exit 1 [2026-11-19T10:40:57.868Z] fi [2026-11-19T10:40:57.868Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-11-19T10:40:57.868Z] if [ -z "$installiso" ]; then [2026-11-19T10:40:57.868Z] echo "Unable to detect install iso" [2026-11-19T10:40:57.868Z] echo "Detected isos" [2026-11-19T10:40:57.868Z] echo "$isos" [2026-11-19T10:40:57.868Z] exit 1 [2026-11-19T10:40:57.868Z] fi [2026-11-19T10:40:57.868Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-11-19T10:40:57.868Z] ] *** [2026-11-19T10:41:03.893Z] changed: [an-a01n01] [2026-11-19T10:41:03.893Z] changed: [an-a01n02] [2026-11-19T10:41:03.893Z] [2026-11-19T10:41:03.893Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-11-19T10:41:03.893Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-11-19T10:41:03.893Z] [2026-11-19T10:41:03.893Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T10:41:03.893Z] ] *** [2026-11-19T10:41:04.626Z] skipping: [an-a01n02] [2026-11-19T10:41:04.626Z] ok: [an-a01n01] [2026-11-19T10:41:04.626Z] [2026-11-19T10:41:04.626Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 5db653ed-72c0-4a3a-b385-e86c6a727b47 _raw_params=anvil-provision-server -vv --log-secure \ [2026-11-19T10:41:04.626Z] --ci-test \ [2026-11-19T10:41:04.626Z] --name an-test-deploy{{ servernum }} \ [2026-11-19T10:41:04.626Z] --os almalinux9 \ [2026-11-19T10:41:04.626Z] --cpu 2 --ram 4G \ [2026-11-19T10:41:04.626Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-11-19T10:41:04.626Z] --install-media {{ installisouuid }} \ [2026-11-19T10:41:04.626Z] --driver-disc deploy{{ servernum }}.iso [2026-11-19T10:41:04.626Z] ] *** [2026-11-19T10:41:04.626Z] skipping: [an-a01n01] [2026-11-19T10:41:04.626Z] skipping: [an-a01n02] [2026-11-19T10:41:04.626Z] [2026-11-19T10:41:04.626Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-11-19T10:41:04.626Z] skipping: [an-a01n01] [2026-11-19T10:41:04.626Z] skipping: [an-a01n02] [2026-11-19T10:41:04.626Z] [2026-11-19T10:41:04.626Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-11-19T10:41:04.969Z] skipping: [an-a01n01] [2026-11-19T10:41:04.969Z] skipping: [an-a01n02] [2026-11-19T10:41:04.969Z] [2026-11-19T10:41:04.969Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-11-19T10:41:04.969Z] skipping: [an-a01n01] [2026-11-19T10:41:04.969Z] skipping: [an-a01n02] [2026-11-19T10:41:04.969Z] [2026-11-19T10:41:04.969Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-11-19T10:41:04.969Z] skipping: [an-a01n01] [2026-11-19T10:41:04.969Z] skipping: [an-a01n02] [2026-11-19T10:41:04.969Z] [2026-11-19T10:41:04.969Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-11-19T10:41:04.969Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-11-19T10:41:04.969Z] [2026-11-19T10:41:04.969Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:41:04.969Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-11-19T10:41:04.969Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-11-19T10:41:04.969Z] if [ "$numservers" != "$startedservers" ]; then [2026-11-19T10:41:04.969Z] echo "Detected $numservers but only $startedservers in Started state" [2026-11-19T10:41:04.969Z] echo "pcs:" [2026-11-19T10:41:04.969Z] echo "$pcsstatus" [2026-11-19T10:41:04.969Z] exit 1 [2026-11-19T10:41:04.969Z] fi [2026-11-19T10:41:04.969Z] exit 0 [2026-11-19T10:41:04.969Z] ] *** [2026-11-19T10:41:07.291Z] ok: [an-a01n02] [2026-11-19T10:41:07.291Z] ok: [an-a01n01] [2026-11-19T10:41:07.291Z] [2026-11-19T10:41:07.291Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-11-19T10:41:07.291Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-11-19T10:41:07.291Z] servers="$(echo "$hypervisor" | wc -l)" [2026-11-19T10:41:07.291Z] ret=0 [2026-11-19T10:41:07.291Z] if [ "$servers" != "1" ]; then [2026-11-19T10:41:07.291Z] echo "Not all servers are running on the same node!" [2026-11-19T10:41:07.291Z] echo "$pcsstatus" [2026-11-19T10:41:07.291Z] ret=1 [2026-11-19T10:41:07.291Z] fi [2026-11-19T10:41:07.291Z] echo "$hypervisor" [2026-11-19T10:41:07.291Z] exit $ret [2026-11-19T10:41:07.291Z] ] *** [2026-11-19T10:41:09.001Z] ok: [an-a01n02] [2026-11-19T10:41:09.001Z] ok: [an-a01n01] [2026-11-19T10:41:09.001Z] [2026-11-19T10:41:09.001Z] PLAY [enable upgrade repo] ***************************************************** [2026-11-19T10:41:09.001Z] [2026-11-19T10:41:09.001Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:41:11.321Z] ok: [an-a01n01] [2026-11-19T10:41:11.321Z] ok: [an-a01n02] [2026-11-19T10:41:11.321Z] ok: [an-striker01] [2026-11-19T10:41:11.321Z] ok: [an-striker02] [2026-11-19T10:41:11.321Z] [2026-11-19T10:41:11.321Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-11-19T10:41:11.321Z] dnf config-manager --set-enabled $updaterepo [2026-11-19T10:41:11.321Z] ] *** [2026-11-19T10:41:24.610Z] changed: [an-a01n02] [2026-11-19T10:41:24.610Z] changed: [an-a01n01] [2026-11-19T10:41:24.610Z] changed: [an-striker01] [2026-11-19T10:41:24.610Z] changed: [an-striker02] [2026-11-19T10:41:24.610Z] [2026-11-19T10:41:24.610Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-11-19T10:41:24.952Z] ok: [an-striker01] [2026-11-19T10:41:24.952Z] ok: [an-striker02] [2026-11-19T10:41:24.952Z] ok: [an-a01n01] [2026-11-19T10:41:24.952Z] ok: [an-a01n02] [2026-11-19T10:41:24.952Z] [2026-11-19T10:41:24.952Z] PLAY [start upgrade process] *************************************************** [2026-11-19T10:41:24.952Z] [2026-11-19T10:41:24.952Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T10:41:26.660Z] ok: [an-striker01] [2026-11-19T10:41:26.660Z] [2026-11-19T10:41:26.660Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-11-19T10:41:26.660Z] ] *** [2026-11-19T11:01:11.989Z] changed: [an-striker01] [2026-11-19T11:01:11.989Z] [2026-11-19T11:01:11.989Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-11-19T11:01:45.574Z] changed: [an-striker01] [2026-11-19T11:01:45.574Z] [2026-11-19T11:01:45.574Z] PLAY [reconnect all nodes] ***************************************************** [2026-11-19T11:01:45.574Z] [2026-11-19T11:01:45.574Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:01:46.308Z] ok: [an-a01n01] [2026-11-19T11:01:46.308Z] ok: [an-a01n02] [2026-11-19T11:01:46.308Z] ok: [an-striker02] [2026-11-19T11:01:46.308Z] ok: [an-striker01] [2026-11-19T11:01:46.308Z] [2026-11-19T11:01:46.308Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-11-19T11:01:46.650Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02 [2026-11-19T11:01:46.650Z] [2026-11-19T11:01:46.650Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-11-19T11:05:10.880Z] ok: [an-a01n01 -> localhost] [2026-11-19T11:05:10.880Z] ok: [an-a01n02 -> localhost] [2026-11-19T11:05:10.880Z] ok: [an-striker01 -> localhost] [2026-11-19T11:05:10.880Z] ok: [an-striker02 -> localhost] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-11-19T11:05:10.880Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-11-19T11:05:10.880Z] exit 1 [2026-11-19T11:05:10.880Z] fi [2026-11-19T11:05:10.880Z] ] *** [2026-11-19T11:05:10.880Z] changed: [an-a01n01] [2026-11-19T11:05:10.880Z] changed: [an-a01n02] [2026-11-19T11:05:10.880Z] changed: [an-striker01] [2026-11-19T11:05:10.880Z] changed: [an-striker02] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] PLAY [check all nodes status after upgrade] ************************************ [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:10.880Z] ok: [an-striker01] [2026-11-19T11:05:10.880Z] ok: [an-a01n02] [2026-11-19T11:05:10.880Z] ok: [an-striker02] [2026-11-19T11:05:10.880Z] ok: [an-a01n01] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-11-19T11:05:10.880Z] echo "perl-Log-Journald update not found!" [2026-11-19T11:05:10.880Z] exit 1 [2026-11-19T11:05:10.880Z] fi [2026-11-19T11:05:10.880Z] ] *** [2026-11-19T11:05:10.880Z] changed: [an-a01n01] [2026-11-19T11:05:10.880Z] changed: [an-a01n02] [2026-11-19T11:05:10.880Z] changed: [an-striker01] [2026-11-19T11:05:10.880Z] changed: [an-striker02] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] PLAY [check nodes status after upgrade] **************************************** [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:10.880Z] ok: [an-a01n02] [2026-11-19T11:05:10.880Z] ok: [an-a01n01] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-11-19T11:05:10.880Z] echo "akmod-drbd update not found!" [2026-11-19T11:05:10.880Z] exit 1 [2026-11-19T11:05:10.880Z] fi [2026-11-19T11:05:10.880Z] ] *** [2026-11-19T11:05:10.880Z] changed: [an-a01n02] [2026-11-19T11:05:10.880Z] changed: [an-a01n01] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] PLAY [check hosted node status after upgrade] ********************************** [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:10.880Z] ok: [an-a01n02] [2026-11-19T11:05:10.880Z] ok: [an-a01n01] [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T11:05:10.880Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T11:05:10.880Z] [2026-11-19T11:05:10.880Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T11:05:10.880Z] ] *** [2026-11-19T11:05:12.064Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T11:05:12.064Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T11:05:12.064Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T11:05:12.064Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T11:05:12.064Z] [2026-11-19T11:05:12.064Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-11-19T11:05:12.064Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-11-19T11:05:12.064Z] [2026-11-19T11:05:12.064Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-11-19T11:05:12.064Z] ] *** [2026-11-19T11:05:13.882Z] ok: [an-a01n02] [2026-11-19T11:05:13.882Z] ok: [an-a01n01] [2026-11-19T11:05:13.882Z] [2026-11-19T11:05:13.882Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-11-19T11:05:13.882Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-11-19T11:05:13.882Z] [2026-11-19T11:05:13.882Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-11-19T11:05:13.882Z] ] *** [2026-11-19T11:05:14.224Z] ok: [an-a01n02] [2026-11-19T11:05:14.224Z] ok: [an-a01n01] [2026-11-19T11:05:14.224Z] [2026-11-19T11:05:14.224Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-11-19T11:05:14.224Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-11-19T11:05:14.224Z] [2026-11-19T11:05:14.224Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-11-19T11:05:14.224Z] ] *** [2026-11-19T11:05:14.957Z] ok: [an-a01n01] [2026-11-19T11:05:14.957Z] ok: [an-a01n02] [2026-11-19T11:05:14.957Z] [2026-11-19T11:05:14.957Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-11-19T11:05:14.957Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-11-19T11:05:14.957Z] [2026-11-19T11:05:14.957Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-11-19T11:05:14.957Z] ] *** [2026-11-19T11:05:16.144Z] ok: [an-a01n02] [2026-11-19T11:05:16.144Z] ok: [an-a01n01] [2026-11-19T11:05:16.144Z] [2026-11-19T11:05:16.144Z] PLAY [test anvil package self update] ****************************************** [2026-11-19T11:05:16.144Z] [2026-11-19T11:05:16.144Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:18.471Z] ok: [an-a01n02] [2026-11-19T11:05:18.471Z] ok: [an-a01n01] [2026-11-19T11:05:18.471Z] ok: [an-striker02] [2026-11-19T11:05:18.471Z] ok: [an-striker01] [2026-11-19T11:05:18.471Z] [2026-11-19T11:05:18.471Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-11-19T11:05:20.789Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-11-19T11:05:20.789Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-11-19T11:05:20.789Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-11-19T11:05:20.789Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-11-19T11:05:20.789Z] changed: [an-a01n02] => (item=scancore.service) [2026-11-19T11:05:20.789Z] changed: [an-a01n01] => (item=scancore.service) [2026-11-19T11:05:20.789Z] changed: [an-striker02] => (item=scancore.service) [2026-11-19T11:05:20.789Z] changed: [an-striker01] => (item=scancore.service) [2026-11-19T11:05:20.789Z] [2026-11-19T11:05:20.789Z] PLAY [test striker package self update] **************************************** [2026-11-19T11:05:20.789Z] [2026-11-19T11:05:20.789Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:22.500Z] ok: [an-striker02] [2026-11-19T11:05:22.500Z] ok: [an-striker01] [2026-11-19T11:05:22.500Z] [2026-11-19T11:05:22.500Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-11-19T11:05:22.500Z] ] *** [2026-11-19T11:05:50.574Z] changed: [an-striker01] [2026-11-19T11:05:50.574Z] changed: [an-striker02] [2026-11-19T11:05:50.574Z] [2026-11-19T11:05:50.574Z] PLAY [test node package self update] ******************************************* [2026-11-19T11:05:50.574Z] [2026-11-19T11:05:50.574Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:05:50.574Z] ok: [an-a01n01] [2026-11-19T11:05:50.574Z] ok: [an-a01n02] [2026-11-19T11:05:50.574Z] [2026-11-19T11:05:50.574Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-11-19T11:05:50.574Z] ] *** [2026-11-19T11:06:13.830Z] changed: [an-a01n02] [2026-11-19T11:06:13.830Z] changed: [an-a01n01] [2026-11-19T11:06:13.830Z] [2026-11-19T11:06:13.830Z] PLAY [test dr package self update] ********************************************* [2026-11-19T11:06:13.830Z] skipping: no hosts matched [2026-11-19T11:06:13.830Z] [2026-11-19T11:06:13.830Z] PLAY [test anvil package self update] ****************************************** [2026-11-19T11:06:13.830Z] [2026-11-19T11:06:13.830Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:06:15.015Z] ok: [an-striker02] [2026-11-19T11:06:15.015Z] ok: [an-striker01] [2026-11-19T11:06:15.015Z] ok: [an-a01n02] [2026-11-19T11:06:15.015Z] ok: [an-a01n01] [2026-11-19T11:06:15.015Z] [2026-11-19T11:06:15.015Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-11-19T11:06:17.335Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-11-19T11:06:17.335Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-11-19T11:06:17.335Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-11-19T11:06:17.335Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-11-19T11:06:17.335Z] changed: [an-a01n01] => (item=scancore.service) [2026-11-19T11:06:17.335Z] changed: [an-a01n02] => (item=scancore.service) [2026-11-19T11:06:17.335Z] changed: [an-striker01] => (item=scancore.service) [2026-11-19T11:06:17.335Z] changed: [an-striker02] => (item=scancore.service) [2026-11-19T11:06:17.335Z] [2026-11-19T11:06:17.335Z] PLAY [delete server tests] ***************************************************** [2026-11-19T11:06:17.335Z] [2026-11-19T11:06:17.335Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:06:19.656Z] ok: [an-a01n02] [2026-11-19T11:06:19.656Z] ok: [an-a01n01] [2026-11-19T11:06:19.656Z] [2026-11-19T11:06:19.656Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-11-19T11:06:19.656Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-11-19T11:06:19.656Z] [2026-11-19T11:06:19.656Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-11-19T11:06:19.656Z] ] *** [2026-11-19T11:06:23.551Z] ok: [an-a01n02] => (item=an-a01n01) [2026-11-19T11:06:23.551Z] ok: [an-a01n01] => (item=an-a01n01) [2026-11-19T11:06:23.551Z] ok: [an-a01n02] => (item=an-a01n02) [2026-11-19T11:06:23.551Z] ok: [an-a01n01] => (item=an-a01n02) [2026-11-19T11:06:23.551Z] [2026-11-19T11:06:23.551Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-11-19T11:06:23.551Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-11-19T11:06:23.551Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-11-19T11:06:23.551Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-11-19T11:06:23.551Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-11-19T11:06:23.551Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-11-19T11:06:23.551Z] [2026-11-19T11:06:23.551Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:06:23.551Z] ] *** [2026-11-19T11:06:25.263Z] skipping: [an-a01n01] [2026-11-19T11:06:25.263Z] ok: [an-a01n02] [2026-11-19T11:06:25.263Z] [2026-11-19T11:06:25.263Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T11:06:25.263Z] --force \ [2026-11-19T11:06:25.263Z] --server an-test-deploy{{ servernum }} [2026-11-19T11:06:25.263Z] ] *** [2026-11-19T11:06:31.283Z] skipping: [an-a01n01] [2026-11-19T11:06:31.283Z] changed: [an-a01n02] [2026-11-19T11:06:31.283Z] [2026-11-19T11:06:31.283Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:06:31.283Z] ] *** [2026-11-19T11:07:19.784Z] skipping: [an-a01n01] [2026-11-19T11:07:19.784Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-11-19T11:07:19.784Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-11-19T11:07:19.784Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-11-19T11:07:19.784Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-11-19T11:07:19.784Z] ok: [an-a01n02] [2026-11-19T11:07:19.784Z] [2026-11-19T11:07:19.784Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T11:07:19.784Z] ] *** [2026-11-19T11:07:19.784Z] skipping: [an-a01n01] [2026-11-19T11:07:19.784Z] ok: [an-a01n02] [2026-11-19T11:07:19.784Z] [2026-11-19T11:07:19.784Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T11:07:19.784Z] ] *** [2026-11-19T11:07:20.517Z] skipping: [an-a01n01] [2026-11-19T11:07:20.517Z] ok: [an-a01n02] [2026-11-19T11:07:20.517Z] [2026-11-19T11:07:20.517Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:20.517Z] ] *** [2026-11-19T11:07:22.226Z] skipping: [an-a01n01] [2026-11-19T11:07:22.226Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.932756", "end": "2026-11-19 06:07:21.776509", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 06:07:20.843753", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T11:07:22.226Z] ...ignoring [2026-11-19T11:07:22.226Z] [2026-11-19T11:07:22.226Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T11:07:22.226Z] --force \ [2026-11-19T11:07:22.226Z] --server an-test-deploy{{ servernum }} [2026-11-19T11:07:22.226Z] ] *** [2026-11-19T11:07:22.226Z] skipping: [an-a01n01] [2026-11-19T11:07:22.226Z] skipping: [an-a01n02] [2026-11-19T11:07:22.226Z] [2026-11-19T11:07:22.226Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:22.226Z] ] *** [2026-11-19T11:07:22.226Z] skipping: [an-a01n01] [2026-11-19T11:07:22.226Z] skipping: [an-a01n02] [2026-11-19T11:07:22.226Z] [2026-11-19T11:07:22.226Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T11:07:22.226Z] ] *** [2026-11-19T11:07:22.226Z] skipping: [an-a01n01] [2026-11-19T11:07:22.226Z] skipping: [an-a01n02] [2026-11-19T11:07:22.226Z] [2026-11-19T11:07:22.226Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T11:07:22.226Z] ] *** [2026-11-19T11:07:22.226Z] skipping: [an-a01n01] [2026-11-19T11:07:22.226Z] skipping: [an-a01n02] [2026-11-19T11:07:22.226Z] [2026-11-19T11:07:22.226Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:22.226Z] ] *** [2026-11-19T11:07:23.936Z] skipping: [an-a01n01] [2026-11-19T11:07:23.936Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.906719", "end": "2026-11-19 06:07:23.547379", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 06:07:22.640660", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T11:07:23.936Z] ...ignoring [2026-11-19T11:07:23.936Z] [2026-11-19T11:07:23.936Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T11:07:23.936Z] --force \ [2026-11-19T11:07:23.936Z] --server an-test-deploy{{ servernum }} [2026-11-19T11:07:23.936Z] ] *** [2026-11-19T11:07:23.936Z] skipping: [an-a01n01] [2026-11-19T11:07:23.936Z] skipping: [an-a01n02] [2026-11-19T11:07:23.936Z] [2026-11-19T11:07:23.936Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:23.936Z] ] *** [2026-11-19T11:07:23.936Z] skipping: [an-a01n01] [2026-11-19T11:07:23.936Z] skipping: [an-a01n02] [2026-11-19T11:07:23.936Z] [2026-11-19T11:07:23.936Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T11:07:23.936Z] ] *** [2026-11-19T11:07:23.936Z] skipping: [an-a01n01] [2026-11-19T11:07:23.936Z] skipping: [an-a01n02] [2026-11-19T11:07:23.936Z] [2026-11-19T11:07:23.936Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T11:07:23.936Z] ] *** [2026-11-19T11:07:23.936Z] skipping: [an-a01n01] [2026-11-19T11:07:23.936Z] skipping: [an-a01n02] [2026-11-19T11:07:23.936Z] [2026-11-19T11:07:23.936Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:23.936Z] ] *** [2026-11-19T11:07:25.647Z] skipping: [an-a01n01] [2026-11-19T11:07:25.647Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.880840", "end": "2026-11-19 06:07:25.372589", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 06:07:24.491749", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T11:07:25.647Z] ...ignoring [2026-11-19T11:07:25.647Z] [2026-11-19T11:07:25.647Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T11:07:25.647Z] --force \ [2026-11-19T11:07:25.647Z] --server an-test-deploy{{ servernum }} [2026-11-19T11:07:25.647Z] ] *** [2026-11-19T11:07:25.647Z] skipping: [an-a01n01] [2026-11-19T11:07:25.647Z] skipping: [an-a01n02] [2026-11-19T11:07:25.647Z] [2026-11-19T11:07:25.647Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:25.647Z] ] *** [2026-11-19T11:07:25.647Z] skipping: [an-a01n01] [2026-11-19T11:07:25.647Z] skipping: [an-a01n02] [2026-11-19T11:07:25.647Z] [2026-11-19T11:07:25.647Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T11:07:25.647Z] ] *** [2026-11-19T11:07:25.989Z] skipping: [an-a01n01] [2026-11-19T11:07:25.989Z] skipping: [an-a01n02] [2026-11-19T11:07:25.989Z] [2026-11-19T11:07:25.989Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T11:07:25.989Z] ] *** [2026-11-19T11:07:25.989Z] skipping: [an-a01n01] [2026-11-19T11:07:25.989Z] skipping: [an-a01n02] [2026-11-19T11:07:25.989Z] [2026-11-19T11:07:25.989Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:25.989Z] ] *** [2026-11-19T11:07:27.699Z] skipping: [an-a01n01] [2026-11-19T11:07:27.699Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.887313", "end": "2026-11-19 06:07:27.132793", "msg": "non-zero return code", "rc": 1, "start": "2026-11-19 06:07:26.245480", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-11-19T11:07:27.699Z] ...ignoring [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-11-19T11:07:27.699Z] --force \ [2026-11-19T11:07:27.699Z] --server an-test-deploy{{ servernum }} [2026-11-19T11:07:27.699Z] ] *** [2026-11-19T11:07:27.699Z] skipping: [an-a01n01] [2026-11-19T11:07:27.699Z] skipping: [an-a01n02] [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-11-19T11:07:27.699Z] ] *** [2026-11-19T11:07:27.699Z] skipping: [an-a01n01] [2026-11-19T11:07:27.699Z] skipping: [an-a01n02] [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-11-19T11:07:27.699Z] ] *** [2026-11-19T11:07:27.699Z] skipping: [an-a01n01] [2026-11-19T11:07:27.699Z] skipping: [an-a01n02] [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-11-19T11:07:27.699Z] ] *** [2026-11-19T11:07:27.699Z] skipping: [an-a01n01] [2026-11-19T11:07:27.699Z] skipping: [an-a01n02] [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] PLAY [Check anvil available resources from striker01] ************************** [2026-11-19T11:07:27.699Z] [2026-11-19T11:07:27.699Z] TASK [Gathering Facts ] ******************************************************** [2026-11-19T11:07:30.019Z] ok: [an-striker01] [2026-11-19T11:07:30.019Z] [2026-11-19T11:07:30.019Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-11-19T11:07:30.019Z] sleep 300 [2026-11-19T11:07:30.019Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-11-19T11:07:30.019Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-11-19T11:07:30.019Z] ] *** [2026-11-19T11:12:39.767Z] ok: [an-striker01] [2026-11-19T11:12:39.767Z] [2026-11-19T11:12:39.767Z] PLAY RECAP ********************************************************************* [2026-11-19T11:12:39.767Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-11-19T11:12:39.767Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-11-19T11:12:39.767Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-11-19T11:12:39.767Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-11-19T11:12:39.767Z] [2026-11-19T11:12:39.767Z] tar: Removing leading `/' from member names [2026-11-19T11:12:39.767Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-11-19T11:12:39.767Z] /root/ci-tools [2026-11-19T11:12:39.767Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/1578.tar.gz [2026-11-19T11:12:39.767Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-11-19T11:12:39.767Z] This might take a few seconds. [2026-11-19T11:12:39.767Z] [2026-11-19T11:12:39.767Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-11-19T11:12:39.767Z] 𐄂 Disconnected from Red Hat Insights [2026-11-19T11:12:40.109Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-11-19T11:12:40.109Z] [2026-11-19T11:12:40.109Z] Manage your connected systems: https://red.ht/connector [2026-11-19T11:12:42.427Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-11-19T11:12:42.427Z] [2026-11-19T11:12:42.427Z] 1. Register with subscription-manager [2026-11-19T11:12:42.427Z] # subscription-manager register [2026-11-19T11:12:42.427Z] [2026-11-19T11:12:42.427Z] 2. Register with insights-client [2026-11-19T11:12:42.427Z] # insights-client --register [2026-11-19T11:12:43.744Z] This system is currently not registered. [2026-11-19T11:12:44.085Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-11-19T11:12:44.085Z] This might take a few seconds. [2026-11-19T11:12:44.085Z] [2026-11-19T11:12:44.085Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-11-19T11:12:46.404Z] 𐄂 Disconnected from Red Hat Insights [2026-11-19T11:12:47.589Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-11-19T11:12:47.589Z] [2026-11-19T11:12:47.589Z] Manage your connected systems: https://red.ht/connector [2026-11-19T11:12:49.300Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-11-19T11:12:49.300Z] [2026-11-19T11:12:49.300Z] 1. Register with subscription-manager [2026-11-19T11:12:49.300Z] # subscription-manager register [2026-11-19T11:12:49.300Z] [2026-11-19T11:12:49.300Z] 2. Register with insights-client [2026-11-19T11:12:49.300Z] # insights-client --register [2026-11-19T11:12:50.483Z] This system is currently not registered. [2026-11-19T11:12:50.832Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-11-19T11:12:50.832Z] This might take a few seconds. [2026-11-19T11:12:50.832Z] [2026-11-19T11:12:50.832Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-11-19T11:12:53.169Z] 𐄂 Disconnected from Red Hat Insights [2026-11-19T11:12:53.902Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-11-19T11:12:53.902Z] [2026-11-19T11:12:53.902Z] Manage your connected systems: https://red.ht/connector [2026-11-19T11:12:55.087Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-11-19T11:12:55.087Z] [2026-11-19T11:12:55.087Z] 1. Register with subscription-manager [2026-11-19T11:12:55.087Z] # subscription-manager register [2026-11-19T11:12:55.087Z] [2026-11-19T11:12:55.087Z] 2. Register with insights-client [2026-11-19T11:12:55.087Z] # insights-client --register [2026-11-19T11:12:56.270Z] This system is currently not registered. [2026-11-19T11:12:56.612Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-11-19T11:12:56.612Z] This might take a few seconds. [2026-11-19T11:12:56.612Z] [2026-11-19T11:12:56.612Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-11-19T11:12:58.452Z] 𐄂 Disconnected from Red Hat Insights [2026-11-19T11:12:59.638Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-11-19T11:12:59.638Z] [2026-11-19T11:12:59.638Z] Manage your connected systems: https://red.ht/connector [2026-11-19T11:13:00.824Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-11-19T11:13:00.824Z] [2026-11-19T11:13:00.824Z] 1. Register with subscription-manager [2026-11-19T11:13:00.824Z] # subscription-manager register [2026-11-19T11:13:00.824Z] [2026-11-19T11:13:00.824Z] 2. Register with insights-client [2026-11-19T11:13:00.824Z] # insights-client --register [2026-11-19T11:13:01.557Z] This system is currently not registered. [2026-11-19T11:13:02.289Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-11-19T11:13:02.289Z] This might take a few seconds. [2026-11-19T11:13:02.289Z] [2026-11-19T11:13:02.289Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-11-19T11:13:04.608Z] 𐄂 Disconnected from Red Hat Insights [2026-11-19T11:13:05.340Z] 𐄂 Disconnected from Red Hat Subscription Management [2026-11-19T11:13:05.340Z] [2026-11-19T11:13:05.340Z] Manage your connected systems: https://red.ht/connector [2026-11-19T11:13:06.524Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-11-19T11:13:06.524Z] [2026-11-19T11:13:06.524Z] 1. Register with subscription-manager [2026-11-19T11:13:06.524Z] # subscription-manager register [2026-11-19T11:13:06.524Z] [2026-11-19T11:13:06.524Z] 2. Register with insights-client [2026-11-19T11:13:06.524Z] # insights-client --register [2026-11-19T11:13:07.709Z] This system is currently not registered. [2026-11-19T11:13:07.779Z] [Pipeline] } [2026-11-19T11:13:07.782Z] [Pipeline] // timeout [2026-11-19T11:13:07.786Z] [Pipeline] } [2026-11-19T11:13:07.788Z] [Pipeline] // script [2026-11-19T11:13:07.791Z] [Pipeline] } [2026-11-19T11:13:07.793Z] [Pipeline] // stage [2026-11-19T11:13:07.797Z] [Pipeline] stage [2026-11-19T11:13:07.797Z] [Pipeline] { (Declarative: Post Actions) [2026-11-19T11:13:07.803Z] [Pipeline] cleanWs [2026-11-19T11:13:07.870Z] [WS-CLEANUP] Deleting project workspace... [2026-11-19T11:13:07.870Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-11-19T11:13:07.893Z] [WS-CLEANUP] done [2026-11-19T11:13:07.898Z] [Pipeline] script [2026-11-19T11:13:07.901Z] [Pipeline] { [2026-11-19T11:13:07.917Z] [Pipeline] stage [2026-11-19T11:13:07.917Z] [Pipeline] { (Copying logs) [2026-11-19T11:13:07.952Z] [Pipeline] node [2026-11-19T11:13:08.049Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-11-19T11:13:08.049Z] [Pipeline] { [2026-11-19T11:13:08.085Z] [Pipeline] node [2026-11-19T11:13:08.181Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-11-19T11:13:08.181Z] [Pipeline] { [2026-11-19T11:13:08.185Z] [Pipeline] sh [2026-11-19T11:13:08.440Z] + mkdir -p /var/lib/jenkins/locks [2026-11-19T11:13:08.449Z] [Pipeline] echo [2026-11-19T11:13:08.449Z] RWLock: FD for lock log_archive in stage projectFinishUp is 431 [2026-11-19T11:13:08.452Z] [Pipeline] sleep [2026-11-19T11:13:08.452Z] No need to sleep any longer [2026-11-19T11:13:08.455Z] [Pipeline] echo [2026-11-19T11:13:08.455Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-11-19T11:13:08.458Z] [Pipeline] } [2026-11-19T11:13:08.460Z] [Pipeline] // node [2026-11-19T11:13:08.463Z] [Pipeline] echo [2026-11-19T11:13:08.463Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1578/ [2026-11-19T11:13:08.464Z] [Pipeline] sh [2026-11-19T11:13:08.719Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/1578/ [2026-11-19T11:13:08.720Z] [Pipeline] sh