[2026-07-09T08:43:55.078Z] Started by upstream project "anvil/anvil-build-cron" build number 109 [2026-07-09T08:43:55.078Z] originally caused by: [2026-07-09T08:43:55.078Z] Started by timer [2026-07-09T08:43:56.568Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-07-09T08:43:56.581Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-07-09T08:43:56.808Z] Examining kronosnet/ci-tools [2026-07-09T08:43:56.808Z] Attempting to resolve main as a branch [2026-07-09T08:43:56.951Z] Resolved main as branch main at revision 08eec0b40de1c061b67d067ec1534b5930b1a926 [2026-07-09T08:43:56.952Z] Selected Git installation does not exist. Using Default [2026-07-09T08:43:56.952Z] The recommended git tool is: NONE [2026-07-09T08:43:56.953Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-07-09T08:43:56.956Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-07-09T08:43:56.956Z] Wiping out workspace first. [2026-07-09T08:43:56.980Z] Cloning the remote Git repository [2026-07-09T08:43:56.980Z] Using shallow clone with depth 1 [2026-07-09T08:43:56.980Z] Avoid fetching tags [2026-07-09T08:43:56.980Z] Honoring refspec on initial clone [2026-07-09T08:43:56.981Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-07-09T08:43:56.981Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b # timeout=10 [2026-07-09T08:43:56.991Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-07-09T08:43:56.991Z] > git --version # timeout=10 [2026-07-09T08:43:56.998Z] > git --version # 'git version 2.52.0' [2026-07-09T08:43:56.998Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-07-09T08:43:57.004Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-07-09T08:43:57.439Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-07-09T08:43:57.446Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-07-09T08:43:57.455Z] Avoid second fetch [2026-07-09T08:43:57.455Z] Checking out Revision 08eec0b40de1c061b67d067ec1534b5930b1a926 (main) [2026-07-09T08:43:57.455Z] > git config core.sparsecheckout # timeout=10 [2026-07-09T08:43:57.462Z] > git checkout -f 08eec0b40de1c061b67d067ec1534b5930b1a926 # timeout=10 [2026-07-09T08:43:57.494Z] Commit message: "ansible: reboot nodes after updates are applied" [2026-07-09T08:43:57.504Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-07-09T08:43:57.706Z] Examining kronosnet/ci-tools [2026-07-09T08:43:57.706Z] Attempting to resolve main as a branch [2026-07-09T08:43:57.866Z] Resolved main as branch main at revision 08eec0b40de1c061b67d067ec1534b5930b1a926 [2026-07-09T08:43:57.868Z] Selected Git installation does not exist. Using Default [2026-07-09T08:43:57.868Z] The recommended git tool is: NONE [2026-07-09T08:43:57.869Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-07-09T08:43:57.875Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-07-09T08:43:57.875Z] Wiping out workspace first. [2026-07-09T08:43:57.876Z] Cloning the remote Git repository [2026-07-09T08:43:57.876Z] Using shallow clone with depth 1 [2026-07-09T08:43:57.876Z] Avoid fetching tags [2026-07-09T08:43:57.876Z] Honoring refspec on initial clone [2026-07-09T08:43:57.876Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-07-09T08:43:57.876Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@6 # timeout=10 [2026-07-09T08:43:57.886Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-07-09T08:43:57.886Z] > git --version # timeout=10 [2026-07-09T08:43:57.893Z] > git --version # 'git version 2.52.0' [2026-07-09T08:43:57.893Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-07-09T08:43:57.900Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-07-09T08:43:58.386Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-07-09T08:43:58.395Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-07-09T08:43:58.403Z] Avoid second fetch [2026-07-09T08:43:58.403Z] Checking out Revision 08eec0b40de1c061b67d067ec1534b5930b1a926 (main) [2026-07-09T08:43:58.403Z] > git config core.sparsecheckout # timeout=10 [2026-07-09T08:43:58.410Z] > git checkout -f 08eec0b40de1c061b67d067ec1534b5930b1a926 # timeout=10 [2026-07-09T08:43:58.443Z] Commit message: "ansible: reboot nodes after updates are applied" [2026-07-09T08:43:58.647Z] [Pipeline] Start of Pipeline [2026-07-09T08:43:58.784Z] [Pipeline] node [2026-07-09T08:43:58.791Z] Running on anvil-ci-bm-phy07 in /srv/workspace/anvil/anvil-build-functional-testing [2026-07-09T08:43:58.792Z] [Pipeline] { [2026-07-09T08:43:58.806Z] [Pipeline] stage [2026-07-09T08:43:58.807Z] [Pipeline] { (Prep build env) [2026-07-09T08:43:58.824Z] [Pipeline] script [2026-07-09T08:43:58.834Z] [Pipeline] { [2026-07-09T08:43:58.841Z] [Pipeline] cleanWs [2026-07-09T08:43:58.892Z] [WS-CLEANUP] Deleting project workspace... [2026-07-09T08:43:58.892Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-07-09T08:43:58.917Z] [WS-CLEANUP] done [2026-07-09T08:43:58.942Z] [Pipeline] lock [2026-07-09T08:43:58.946Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-07-09T08:43:58.950Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-07-09T08:43:58.950Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-07-09T08:43:58.950Z] [Pipeline] { [2026-07-09T08:43:58.957Z] [Pipeline] } [2026-07-09T08:43:58.959Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-07-09T08:43:58.962Z] [Pipeline] // lock [2026-07-09T08:43:58.969Z] [Pipeline] } [2026-07-09T08:43:58.971Z] [Pipeline] // script [2026-07-09T08:43:58.974Z] [Pipeline] } [2026-07-09T08:43:58.985Z] [Pipeline] // stage [2026-07-09T08:43:58.988Z] [Pipeline] stage [2026-07-09T08:43:58.989Z] [Pipeline] { (Run functional testing) [2026-07-09T08:43:59.001Z] [Pipeline] script [2026-07-09T08:43:59.003Z] [Pipeline] { [2026-07-09T08:43:59.025Z] [Pipeline] echo [2026-07-09T08:43:59.025Z] runWithTimeout 360 [2026-07-09T08:43:59.026Z] [Pipeline] timeout [2026-07-09T08:43:59.026Z] Timeout set to expire in 6 hr 0 min [2026-07-09T08:43:59.027Z] [Pipeline] { [2026-07-09T08:43:59.033Z] [Pipeline] sh [2026-07-09T08:43:59.512Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm almalinux-9 enterprise-staging none all [2026-07-09T08:43:59.512Z] CITBRANCH: main [2026-07-09T08:43:59.512Z] CITHOME: /root/ci-tools [2026-07-09T08:43:59.512Z] CITCMD: ci-setup-anvil-bm-vm [2026-07-09T08:43:59.512Z] CITCMDARGS: almalinux-9 enterprise-staging none all [2026-07-09T08:43:59.512Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-07-09T08:43:59.512Z] Genering ssh keys and ssh config [2026-07-09T08:44:00.253Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-07-09T08:47:09.357Z] Destroying VMs (if any) [2026-07-09T08:47:09.357Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-07-09T08:47:09.357Z] Genering ssh keys and ssh config [2026-07-09T08:47:09.357Z] Cache already updated [2026-07-09T08:47:09.357Z] Creating nodes disks [2026-07-09T08:47:09.357Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-07-09T08:47:09.357Z] Generating an-test-deployX kickstart files/isos [2026-07-09T08:47:09.357Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-07-09T08:47:09.357Z] Media current: stdio file, overwriteable [2026-07-09T08:47:09.357Z] Media status : is blank [2026-07-09T08:47:09.357Z] Media summary: 0 sessions, 0 data blocks, 0 data, 800g free [2026-07-09T08:47:09.357Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] ISO image produced: 184 sectors [2026-07-09T08:47:09.357Z] Written to medium : 184 sectors at LBA 0 [2026-07-09T08:47:09.357Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-07-09T08:47:09.357Z] Media current: stdio file, overwriteable [2026-07-09T08:47:09.357Z] Media status : is blank [2026-07-09T08:47:09.357Z] Media summary: 0 sessions, 0 data blocks, 0 data, 802g free [2026-07-09T08:47:09.357Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] ISO image produced: 184 sectors [2026-07-09T08:47:09.357Z] Written to medium : 184 sectors at LBA 0 [2026-07-09T08:47:09.357Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-07-09T08:47:09.357Z] Media current: stdio file, overwriteable [2026-07-09T08:47:09.357Z] Media status : is blank [2026-07-09T08:47:09.357Z] Media summary: 0 sessions, 0 data blocks, 0 data, 802g free [2026-07-09T08:47:09.357Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] ISO image produced: 184 sectors [2026-07-09T08:47:09.357Z] Written to medium : 184 sectors at LBA 0 [2026-07-09T08:47:09.357Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-07-09T08:47:09.357Z] Media current: stdio file, overwriteable [2026-07-09T08:47:09.357Z] Media status : is blank [2026-07-09T08:47:09.357Z] Media summary: 0 sessions, 0 data blocks, 0 data, 802g free [2026-07-09T08:47:09.357Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] ISO image produced: 184 sectors [2026-07-09T08:47:09.357Z] Written to medium : 184 sectors at LBA 0 [2026-07-09T08:47:09.357Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-07-09T08:47:09.357Z] Media current: stdio file, overwriteable [2026-07-09T08:47:09.357Z] Media status : is blank [2026-07-09T08:47:09.357Z] Media summary: 0 sessions, 0 data blocks, 0 data, 802g free [2026-07-09T08:47:09.357Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-07-09T08:47:09.357Z] ISO image produced: 184 sectors [2026-07-09T08:47:09.357Z] Written to medium : 184 sectors at LBA 0 [2026-07-09T08:47:09.357Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Generating cloud-init configs [2026-07-09T08:47:09.357Z] Generating cloud-init metadata and seed images [2026-07-09T08:47:09.357Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-07-09T08:47:09.357Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-07-09T08:47:09.357Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-07-09T08:47:09.357Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-07-09T08:47:09.357Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-07-09T08:47:09.357Z] Creating VMs [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Starting install... [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Creating domain... | 00:00 [2026-07-09T08:47:09.357Z] Domain creation completed. [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Starting install... [2026-07-09T08:47:09.357Z] [2026-07-09T08:47:09.357Z] Creating domain... | 00:00 [2026-07-09T08:47:09.357Z] Domain creation completed. [2026-07-09T08:47:10.554Z] [2026-07-09T08:47:10.554Z] Starting install... [2026-07-09T08:47:10.554Z] [2026-07-09T08:47:10.554Z] Creating domain... | 00:00 [2026-07-09T08:47:10.554Z] Domain creation completed. [2026-07-09T08:47:11.752Z] [2026-07-09T08:47:11.752Z] Starting install... [2026-07-09T08:47:11.752Z] [2026-07-09T08:47:11.752Z] Creating domain... | 00:00 [2026-07-09T08:47:11.752Z] Domain creation completed. [2026-07-09T08:47:13.476Z] [2026-07-09T08:47:13.476Z] Starting install... [2026-07-09T08:47:13.476Z] [2026-07-09T08:47:13.476Z] Creating domain... | 00:00 [2026-07-09T08:47:13.476Z] Domain creation completed. [2026-07-09T08:47:13.476Z] Waiting for nodes and cloud-init to complete full init [2026-07-09T08:47:16.530Z] cloud-init status node an-striker01: [2026-07-09T08:47:19.587Z] cloud-init status node an-striker02: [2026-07-09T08:47:22.642Z] cloud-init status node an-a01n01: [2026-07-09T08:47:26.547Z] cloud-init status node an-a01n02: [2026-07-09T08:47:28.887Z] cloud-init status node an-a01dr01: [2026-07-09T08:47:28.887Z] Thu Jul 9 04:47:28 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:47:42.232Z] cloud-init status node an-striker01: running [2026-07-09T08:47:42.232Z] cloud-init status node an-striker02: running [2026-07-09T08:47:42.232Z] cloud-init status node an-a01n01: running [2026-07-09T08:47:42.977Z] cloud-init status node an-a01n02: running [2026-07-09T08:47:44.192Z] cloud-init status node an-a01dr01: running [2026-07-09T08:47:44.192Z] Thu Jul 9 04:47:43 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:47:55.650Z] cloud-init status node an-striker01: running [2026-07-09T08:47:55.650Z] cloud-init status node an-striker02: running [2026-07-09T08:47:56.899Z] cloud-init status node an-a01n01: running [2026-07-09T08:47:57.754Z] cloud-init status node an-a01n02: running [2026-07-09T08:47:58.508Z] cloud-init status node an-a01dr01: running [2026-07-09T08:47:58.508Z] Thu Jul 9 04:47:58 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:48:10.860Z] cloud-init status node an-striker01: running [2026-07-09T08:48:10.860Z] cloud-init status node an-striker02: running [2026-07-09T08:48:11.215Z] cloud-init status node an-a01n01: running [2026-07-09T08:48:11.988Z] cloud-init status node an-a01n02: running [2026-07-09T08:48:13.215Z] cloud-init status node an-a01dr01: running [2026-07-09T08:48:13.215Z] Thu Jul 9 04:48:12 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:48:25.483Z] cloud-init status node an-striker01: running [2026-07-09T08:48:25.483Z] cloud-init status node an-striker02: running [2026-07-09T08:48:25.483Z] cloud-init status node an-a01n01: running [2026-07-09T08:48:26.319Z] cloud-init status node an-a01n02: running [2026-07-09T08:48:27.399Z] cloud-init status node an-a01dr01: running [2026-07-09T08:48:27.399Z] Thu Jul 9 04:48:27 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:48:38.700Z] cloud-init status node an-striker01: running [2026-07-09T08:48:39.454Z] cloud-init status node an-striker02: running [2026-07-09T08:48:40.309Z] cloud-init status node an-a01n01: running [2026-07-09T08:48:41.099Z] cloud-init status node an-a01n02: running [2026-07-09T08:48:41.976Z] cloud-init status node an-a01dr01: running [2026-07-09T08:48:41.976Z] Thu Jul 9 04:48:41 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:48:53.904Z] cloud-init status node an-striker01: running [2026-07-09T08:48:53.904Z] cloud-init status node an-striker02: running [2026-07-09T08:48:59.763Z] cloud-init status node an-a01n01: running [2026-07-09T08:49:00.140Z] cloud-init status node an-a01n02: running [2026-07-09T08:49:01.365Z] cloud-init status node an-a01dr01: running [2026-07-09T08:49:01.365Z] Thu Jul 9 04:49:00 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:49:13.164Z] cloud-init status node an-striker01: running [2026-07-09T08:49:13.164Z] cloud-init status node an-striker02: running [2026-07-09T08:49:13.870Z] cloud-init status node an-a01n01: running [2026-07-09T08:49:14.956Z] cloud-init status node an-a01n02: running [2026-07-09T08:49:15.650Z] cloud-init status node an-a01dr01: running [2026-07-09T08:49:15.650Z] Thu Jul 9 04:49:14 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:49:27.341Z] cloud-init status node an-striker01: running [2026-07-09T08:49:27.341Z] cloud-init status node an-striker02: running [2026-07-09T08:49:27.699Z] cloud-init status node an-a01n01: running [2026-07-09T08:49:28.485Z] cloud-init status node an-a01n02: running [2026-07-09T08:49:29.261Z] cloud-init status node an-a01dr01: running [2026-07-09T08:49:29.261Z] Thu Jul 9 04:49:29 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:49:40.810Z] cloud-init status node an-striker01: running [2026-07-09T08:49:40.810Z] cloud-init status node an-striker02: running [2026-07-09T08:49:41.776Z] cloud-init status node an-a01n01: running [2026-07-09T08:49:42.705Z] cloud-init status node an-a01n02: running [2026-07-09T08:49:43.411Z] cloud-init status node an-a01dr01: running [2026-07-09T08:49:43.411Z] Thu Jul 9 04:49:43 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:49:54.623Z] cloud-init status node an-striker01: running [2026-07-09T08:49:54.988Z] cloud-init status node an-striker02: running [2026-07-09T08:49:56.314Z] cloud-init status node an-a01n01: running [2026-07-09T08:49:57.216Z] cloud-init status node an-a01n02: running [2026-07-09T08:49:57.866Z] cloud-init status node an-a01dr01: running [2026-07-09T08:49:57.866Z] Thu Jul 9 04:49:57 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:50:08.618Z] cloud-init status node an-striker01: running [2026-07-09T08:50:09.708Z] cloud-init status node an-striker02: running [2026-07-09T08:50:10.299Z] cloud-init status node an-a01n01: running [2026-07-09T08:50:16.225Z] cloud-init status node an-a01n02: running [2026-07-09T08:50:17.013Z] cloud-init status node an-a01dr01: running [2026-07-09T08:50:17.013Z] Thu Jul 9 04:50:16 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:50:28.056Z] cloud-init status node an-striker01: running [2026-07-09T08:50:28.487Z] cloud-init status node an-striker02: running [2026-07-09T08:50:29.859Z] cloud-init status node an-a01n01: running [2026-07-09T08:50:30.596Z] cloud-init status node an-a01n02: running [2026-07-09T08:50:30.985Z] cloud-init status node an-a01dr01: running [2026-07-09T08:50:30.985Z] Thu Jul 9 04:50:30 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:50:42.121Z] cloud-init status node an-striker01: running [2026-07-09T08:50:44.088Z] cloud-init status node an-striker02: running [2026-07-09T08:50:44.088Z] cloud-init status node an-a01n01: running [2026-07-09T08:50:44.622Z] cloud-init status node an-a01n02: running [2026-07-09T08:50:45.911Z] cloud-init status node an-a01dr01: running [2026-07-09T08:50:45.911Z] Thu Jul 9 04:50:45 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:50:56.200Z] cloud-init status node an-striker01: running [2026-07-09T08:51:02.207Z] cloud-init status node an-striker02: running [2026-07-09T08:51:02.998Z] cloud-init status node an-a01n01: running [2026-07-09T08:51:03.664Z] cloud-init status node an-a01n02: running [2026-07-09T08:51:10.547Z] cloud-init status node an-a01dr01: running [2026-07-09T08:51:10.547Z] Thu Jul 9 04:51:09 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:51:21.302Z] cloud-init status node an-striker01: running [2026-07-09T08:51:21.972Z] cloud-init status node an-striker02: running [2026-07-09T08:51:22.368Z] cloud-init status node an-a01n01: running [2026-07-09T08:51:23.288Z] cloud-init status node an-a01n02: running [2026-07-09T08:51:24.171Z] cloud-init status node an-a01dr01: running [2026-07-09T08:51:24.171Z] Thu Jul 9 04:51:24 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:51:36.524Z] cloud-init status node an-striker01: running [2026-07-09T08:51:36.524Z] cloud-init status node an-striker02: running [2026-07-09T08:51:36.893Z] cloud-init status node an-a01n01: running [2026-07-09T08:51:38.094Z] cloud-init status node an-a01n02: running [2026-07-09T08:51:38.894Z] cloud-init status node an-a01dr01: running [2026-07-09T08:51:38.894Z] Thu Jul 9 04:51:38 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:51:50.021Z] cloud-init status node an-striker01: running [2026-07-09T08:51:51.128Z] cloud-init status node an-striker02: running [2026-07-09T08:51:51.633Z] cloud-init status node an-a01n01: running [2026-07-09T08:51:53.001Z] cloud-init status node an-a01n02: running [2026-07-09T08:51:53.623Z] cloud-init status node an-a01dr01: running [2026-07-09T08:51:53.623Z] Thu Jul 9 04:51:53 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:52:05.810Z] cloud-init status node an-striker01: running [2026-07-09T08:52:05.810Z] cloud-init status node an-striker02: running [2026-07-09T08:52:06.585Z] cloud-init status node an-a01n01: running [2026-07-09T08:52:07.372Z] cloud-init status node an-a01n02: running [2026-07-09T08:52:08.147Z] cloud-init status node an-a01dr01: running [2026-07-09T08:52:08.147Z] Thu Jul 9 04:52:07 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:52:19.867Z] cloud-init status node an-striker01: running [2026-07-09T08:52:20.265Z] cloud-init status node an-striker02: running [2026-07-09T08:52:21.548Z] cloud-init status node an-a01n01: running [2026-07-09T08:52:22.322Z] cloud-init status node an-a01n02: running [2026-07-09T08:52:23.212Z] cloud-init status node an-a01dr01: running [2026-07-09T08:52:23.212Z] Thu Jul 9 04:52:23 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:52:34.773Z] cloud-init status node an-striker01: running [2026-07-09T08:52:35.127Z] cloud-init status node an-striker02: running [2026-07-09T08:52:35.893Z] cloud-init status node an-a01n01: running [2026-07-09T08:52:37.176Z] cloud-init status node an-a01n02: running [2026-07-09T08:52:37.938Z] cloud-init status node an-a01dr01: running [2026-07-09T08:52:37.938Z] Thu Jul 9 04:52:37 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:52:49.111Z] cloud-init status node an-striker01: running [2026-07-09T08:52:49.870Z] cloud-init status node an-striker02: running [2026-07-09T08:52:51.085Z] cloud-init status node an-a01n01: running [2026-07-09T08:52:51.848Z] cloud-init status node an-a01n02: running [2026-07-09T08:52:53.071Z] cloud-init status node an-a01dr01: running [2026-07-09T08:52:53.071Z] Thu Jul 9 04:52:52 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:53:04.182Z] cloud-init status node an-striker01: running [2026-07-09T08:53:04.932Z] cloud-init status node an-striker02: running [2026-07-09T08:53:05.676Z] cloud-init status node an-a01n01: running [2026-07-09T08:53:06.872Z] cloud-init status node an-a01n02: running [2026-07-09T08:53:07.614Z] cloud-init status node an-a01dr01: running [2026-07-09T08:53:07.614Z] Thu Jul 9 04:53:07 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:53:18.669Z] cloud-init status node an-striker01: running [2026-07-09T08:53:19.022Z] cloud-init status node an-striker02: running [2026-07-09T08:53:20.256Z] cloud-init status node an-a01n01: running [2026-07-09T08:53:21.012Z] cloud-init status node an-a01n02: running [2026-07-09T08:53:21.819Z] cloud-init status node an-a01dr01: running [2026-07-09T08:53:21.819Z] Thu Jul 9 04:53:21 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:53:32.988Z] cloud-init status node an-striker01: running [2026-07-09T08:53:33.336Z] cloud-init status node an-striker02: running [2026-07-09T08:53:34.534Z] cloud-init status node an-a01n01: running [2026-07-09T08:53:35.275Z] cloud-init status node an-a01n02: running [2026-07-09T08:53:36.473Z] cloud-init status node an-a01dr01: running [2026-07-09T08:53:36.473Z] Thu Jul 9 04:53:36 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:53:47.542Z] cloud-init status node an-striker01: running [2026-07-09T08:53:48.286Z] cloud-init status node an-striker02: running [2026-07-09T08:53:49.026Z] cloud-init status node an-a01n01: running [2026-07-09T08:53:50.223Z] cloud-init status node an-a01n02: running [2026-07-09T08:53:50.965Z] cloud-init status node an-a01dr01: running [2026-07-09T08:53:50.965Z] Thu Jul 9 04:53:50 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:54:02.031Z] cloud-init status node an-striker01: running [2026-07-09T08:54:02.771Z] cloud-init status node an-striker02: running [2026-07-09T08:54:03.967Z] cloud-init status node an-a01n01: running [2026-07-09T08:54:04.710Z] cloud-init status node an-a01n02: running [2026-07-09T08:54:05.451Z] cloud-init status node an-a01dr01: running [2026-07-09T08:54:05.451Z] Thu Jul 9 04:54:05 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:54:16.493Z] cloud-init status node an-striker01: running [2026-07-09T08:54:17.239Z] cloud-init status node an-striker02: running [2026-07-09T08:54:18.439Z] cloud-init status node an-a01n01: running [2026-07-09T08:54:19.180Z] cloud-init status node an-a01n02: running [2026-07-09T08:54:20.377Z] cloud-init status node an-a01dr01: running [2026-07-09T08:54:20.377Z] Thu Jul 9 04:54:20 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:54:31.398Z] cloud-init status node an-striker01: running [2026-07-09T08:54:32.140Z] cloud-init status node an-striker02: running [2026-07-09T08:54:32.880Z] cloud-init status node an-a01n01: running [2026-07-09T08:54:34.078Z] cloud-init status node an-a01n02: running [2026-07-09T08:54:34.820Z] cloud-init status node an-a01dr01: running [2026-07-09T08:54:34.820Z] Thu Jul 9 04:54:34 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:54:45.842Z] cloud-init status node an-striker01: running [2026-07-09T08:54:47.040Z] cloud-init status node an-striker02: running [2026-07-09T08:54:47.782Z] cloud-init status node an-a01n01: running [2026-07-09T08:54:48.982Z] cloud-init status node an-a01n02: running [2026-07-09T08:54:49.723Z] cloud-init status node an-a01dr01: running [2026-07-09T08:54:49.723Z] Thu Jul 9 04:54:49 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:55:00.744Z] cloud-init status node an-striker01: running [2026-07-09T08:55:01.486Z] cloud-init status node an-striker02: running [2026-07-09T08:55:02.682Z] cloud-init status node an-a01n01: running [2026-07-09T08:55:03.424Z] cloud-init status node an-a01n02: running [2026-07-09T08:55:04.621Z] cloud-init status node an-a01dr01: running [2026-07-09T08:55:04.621Z] Thu Jul 9 04:55:04 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:55:15.645Z] cloud-init status node an-striker01: running [2026-07-09T08:55:16.386Z] cloud-init status node an-striker02: running [2026-07-09T08:55:17.584Z] cloud-init status node an-a01n01: running [2026-07-09T08:55:18.327Z] cloud-init status node an-a01n02: running [2026-07-09T08:55:19.526Z] cloud-init status node an-a01dr01: running [2026-07-09T08:55:19.526Z] Thu Jul 9 04:55:19 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:55:30.549Z] cloud-init status node an-striker01: running [2026-07-09T08:55:31.289Z] cloud-init status node an-striker02: running [2026-07-09T08:55:32.487Z] cloud-init status node an-a01n01: running [2026-07-09T08:55:33.228Z] cloud-init status node an-a01n02: running [2026-07-09T08:55:34.428Z] cloud-init status node an-a01dr01: running [2026-07-09T08:55:34.429Z] Thu Jul 9 04:55:34 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:55:45.445Z] cloud-init status node an-striker01: running [2026-07-09T08:55:46.187Z] cloud-init status node an-striker02: running [2026-07-09T08:55:47.382Z] cloud-init status node an-a01n01: running [2026-07-09T08:55:48.125Z] cloud-init status node an-a01n02: running [2026-07-09T08:55:49.321Z] cloud-init status node an-a01dr01: running [2026-07-09T08:55:49.321Z] Thu Jul 9 04:55:48 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:56:00.348Z] cloud-init status node an-striker01: running [2026-07-09T08:56:01.091Z] cloud-init status node an-striker02: running [2026-07-09T08:56:01.831Z] cloud-init status node an-a01n01: running [2026-07-09T08:56:03.029Z] cloud-init status node an-a01n02: running [2026-07-09T08:56:03.772Z] cloud-init status node an-a01dr01: running [2026-07-09T08:56:03.772Z] Thu Jul 9 04:56:03 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:56:14.795Z] cloud-init status node an-striker01: running [2026-07-09T08:56:15.538Z] cloud-init status node an-striker02: running [2026-07-09T08:56:16.736Z] cloud-init status node an-a01n01: running [2026-07-09T08:56:17.480Z] cloud-init status node an-a01n02: running [2026-07-09T08:56:18.678Z] cloud-init status node an-a01dr01: running [2026-07-09T08:56:18.678Z] Thu Jul 9 04:56:18 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:56:29.753Z] cloud-init status node an-striker01: running [2026-07-09T08:56:30.495Z] cloud-init status node an-striker02: running [2026-07-09T08:56:31.238Z] cloud-init status node an-a01n01: running [2026-07-09T08:56:32.435Z] cloud-init status node an-a01n02: running [2026-07-09T08:56:33.177Z] cloud-init status node an-a01dr01: running [2026-07-09T08:56:33.177Z] Thu Jul 9 04:56:33 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:56:44.202Z] cloud-init status node an-striker01: running [2026-07-09T08:56:45.400Z] cloud-init status node an-striker02: running [2026-07-09T08:56:46.142Z] cloud-init status node an-a01n01: running [2026-07-09T08:56:46.893Z] cloud-init status node an-a01n02: running [2026-07-09T08:56:48.094Z] cloud-init status node an-a01dr01: running [2026-07-09T08:56:48.094Z] Thu Jul 9 04:56:47 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:57:01.489Z] cloud-init status node an-striker01: running [2026-07-09T08:57:01.489Z] cloud-init status node an-striker02: running [2026-07-09T08:57:01.835Z] cloud-init status node an-a01n01: running [2026-07-09T08:57:03.032Z] cloud-init status node an-a01n02: running [2026-07-09T08:57:04.231Z] cloud-init status node an-a01dr01: running [2026-07-09T08:57:04.231Z] Thu Jul 9 04:57:03 EDT 2026 actual: 0 expected: 5 [2026-07-09T08:57:17.635Z] cloud-init status node an-striker01: running [2026-07-09T08:57:18.377Z] cloud-init status node an-striker02: running [2026-07-09T08:57:20.781Z] cloud-init status node an-a01n01: done [2026-07-09T08:57:22.007Z] cloud-init status node an-a01n02: running [2026-07-09T08:57:23.206Z] cloud-init status node an-a01dr01: done [2026-07-09T08:57:23.206Z] Thu Jul 9 04:57:22 EDT 2026 actual: 2 expected: 5 [2026-07-09T08:57:34.319Z] cloud-init status node an-striker01: running [2026-07-09T08:57:35.532Z] cloud-init status node an-striker02: running [2026-07-09T08:57:36.731Z] cloud-init status node an-a01n01: done [2026-07-09T08:57:37.481Z] cloud-init status node an-a01n02: done [2026-07-09T08:57:38.839Z] cloud-init status node an-a01dr01: done [2026-07-09T08:57:38.839Z] Thu Jul 9 04:57:38 EDT 2026 actual: 3 expected: 5 [2026-07-09T08:57:49.905Z] cloud-init status node an-striker01: running [2026-07-09T08:57:50.650Z] cloud-init status node an-striker02: running [2026-07-09T08:57:51.849Z] cloud-init status node an-a01n01: done [2026-07-09T08:57:52.590Z] cloud-init status node an-a01n02: done [2026-07-09T08:57:53.405Z] cloud-init status node an-a01dr01: done [2026-07-09T08:57:53.405Z] Thu Jul 9 04:57:53 EDT 2026 actual: 3 expected: 5 [2026-07-09T08:58:06.773Z] cloud-init status node an-striker01: running [2026-07-09T08:58:06.773Z] cloud-init status node an-striker02: running [2026-07-09T08:58:07.519Z] cloud-init status node an-a01n01: done [2026-07-09T08:58:08.575Z] cloud-init status node an-a01n02: done [2026-07-09T08:58:09.317Z] cloud-init status node an-a01dr01: done [2026-07-09T08:58:09.317Z] Thu Jul 9 04:58:09 EDT 2026 actual: 3 expected: 5 [2026-07-09T08:58:20.350Z] cloud-init status node an-striker01: running [2026-07-09T08:58:21.548Z] cloud-init status node an-striker02: running [2026-07-09T08:58:22.291Z] cloud-init status node an-a01n01: done [2026-07-09T08:58:23.491Z] cloud-init status node an-a01n02: done [2026-07-09T08:58:24.239Z] cloud-init status node an-a01dr01: done [2026-07-09T08:58:24.239Z] Thu Jul 9 04:58:23 EDT 2026 actual: 3 expected: 5 [2026-07-09T08:58:37.599Z] cloud-init status node an-striker01: done [2026-07-09T08:58:39.329Z] cloud-init status node an-striker02: done [2026-07-09T08:58:40.072Z] cloud-init status node an-a01n01: done [2026-07-09T08:58:41.273Z] cloud-init status node an-a01n02: done [2026-07-09T08:58:42.067Z] cloud-init status node an-a01dr01: done [2026-07-09T08:58:42.068Z] Thu Jul 9 04:58:41 EDT 2026 actual: 5 expected: 5 [2026-07-09T08:58:42.068Z] Building simengine model [2026-07-09T08:58:55.444Z] Generating anvil striker auto config [2026-07-09T08:58:58.634Z] Cloning into 'anvil-test-suite'... [2026-07-09T08:58:59.380Z] generated ansible config: [2026-07-09T08:58:59.380Z] ========================= [2026-07-09T08:58:59.380Z] [defaults] [2026-07-09T08:58:59.380Z] inventory = ansible-inventory [2026-07-09T08:58:59.380Z] forks = 500 [2026-07-09T08:58:59.380Z] any_errors_fatal = true [2026-07-09T08:58:59.380Z] display_args_to_stdout = true [2026-07-09T08:58:59.380Z] [2026-07-09T08:58:59.380Z] [ssh_connection] [2026-07-09T08:58:59.380Z] retries = 20 [2026-07-09T08:58:59.380Z] ========================= [2026-07-09T08:58:59.380Z] [2026-07-09T08:58:59.380Z] generated ansible inventory: [2026-07-09T08:58:59.380Z] ============================ [2026-07-09T08:58:59.380Z] [strikers] [2026-07-09T08:58:59.380Z] an-striker01 [2026-07-09T08:58:59.380Z] an-striker02 [2026-07-09T08:58:59.380Z] [2026-07-09T08:58:59.380Z] [nodes] [2026-07-09T08:58:59.380Z] an-a01n01 [2026-07-09T08:58:59.380Z] an-a01n02 [2026-07-09T08:58:59.380Z] [2026-07-09T08:58:59.380Z] [drnodes] [2026-07-09T08:58:59.380Z] ============================ [2026-07-09T08:58:59.380Z] [2026-07-09T08:58:59.380Z] Checking connectivity to nodes [2026-07-09T08:58:59.380Z] [2026-07-09T08:59:01.755Z] an-a01n01 | SUCCESS => { [2026-07-09T08:59:01.755Z] "ansible_facts": { [2026-07-09T08:59:01.755Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-07-09T08:59:01.755Z] }, [2026-07-09T08:59:01.755Z] "changed": false, [2026-07-09T08:59:01.755Z] "ping": "pong" [2026-07-09T08:59:01.755Z] } [2026-07-09T08:59:01.755Z] an-a01n02 | SUCCESS => { [2026-07-09T08:59:01.755Z] "ansible_facts": { [2026-07-09T08:59:01.755Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-07-09T08:59:01.755Z] }, [2026-07-09T08:59:01.755Z] "changed": false, [2026-07-09T08:59:01.755Z] "ping": "pong" [2026-07-09T08:59:01.755Z] } [2026-07-09T08:59:01.755Z] an-striker02 | SUCCESS => { [2026-07-09T08:59:01.755Z] "ansible_facts": { [2026-07-09T08:59:01.755Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-07-09T08:59:01.755Z] }, [2026-07-09T08:59:01.755Z] "changed": false, [2026-07-09T08:59:01.755Z] "ping": "pong" [2026-07-09T08:59:01.755Z] } [2026-07-09T08:59:01.755Z] an-striker01 | SUCCESS => { [2026-07-09T08:59:01.755Z] "ansible_facts": { [2026-07-09T08:59:01.755Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-07-09T08:59:01.755Z] }, [2026-07-09T08:59:01.755Z] "changed": false, [2026-07-09T08:59:01.755Z] "ping": "pong" [2026-07-09T08:59:01.755Z] } [2026-07-09T08:59:01.755Z] [2026-07-09T08:59:02.550Z] [2026-07-09T08:59:02.550Z] PLAY [Setup all nodes] ********************************************************* [2026-07-09T08:59:02.550Z] [2026-07-09T08:59:02.550Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T08:59:04.894Z] ok: [an-a01n01] [2026-07-09T08:59:04.894Z] ok: [an-a01n02] [2026-07-09T08:59:04.894Z] ok: [an-striker01] [2026-07-09T08:59:04.894Z] ok: [an-striker02] [2026-07-09T08:59:04.894Z] [2026-07-09T08:59:04.894Z] TASK [create /opt/MegaRAID/storcli/ path=/opt/MegaRAID/storcli/, state=directory, mode=493] *** [2026-07-09T08:59:06.095Z] changed: [an-a01n02] [2026-07-09T08:59:06.095Z] changed: [an-a01n01] [2026-07-09T08:59:06.095Z] changed: [an-striker02] [2026-07-09T08:59:06.095Z] changed: [an-striker01] [2026-07-09T08:59:06.095Z] [2026-07-09T08:59:06.095Z] TASK [wipe old storcli64 path=/opt/MegaRAID/storcli/storcli64, state=absent] *** [2026-07-09T08:59:06.911Z] ok: [an-a01n02] [2026-07-09T08:59:06.911Z] ok: [an-a01n01] [2026-07-09T08:59:06.911Z] ok: [an-striker02] [2026-07-09T08:59:06.911Z] ok: [an-striker01] [2026-07-09T08:59:06.911Z] [2026-07-09T08:59:06.911Z] TASK [get latest Simengine storcli64 url=http://anvil-ci-repo.ci.alteeve.com/ci-repo/storcli64, dest=/opt/MegaRAID/storcli/storcli64, owner=root, group=root, mode=0755] *** [2026-07-09T08:59:08.638Z] changed: [an-a01n01] [2026-07-09T08:59:08.638Z] changed: [an-a01n02] [2026-07-09T08:59:08.638Z] changed: [an-striker02] [2026-07-09T08:59:08.638Z] changed: [an-striker01] [2026-07-09T08:59:08.638Z] [2026-07-09T08:59:08.638Z] PLAY [setup strikers] ********************************************************** [2026-07-09T08:59:08.638Z] [2026-07-09T08:59:08.638Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T08:59:10.379Z] ok: [an-striker01] [2026-07-09T08:59:10.379Z] ok: [an-striker02] [2026-07-09T08:59:10.379Z] [2026-07-09T08:59:10.379Z] TASK [copy striker autogenerated config src={{ deploydir }}/configs/anvil_striker_auto.cfg, dest=/root/anvil_striker_auto.cfg, owner=root, group=root, mode=0644] *** [2026-07-09T08:59:12.721Z] changed: [an-striker01] [2026-07-09T08:59:12.721Z] changed: [an-striker02] [2026-07-09T08:59:12.721Z] [2026-07-09T08:59:12.721Z] PLAY [setup lvm storage] ******************************************************* [2026-07-09T08:59:12.721Z] [2026-07-09T08:59:12.721Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T08:59:13.924Z] ok: [an-a01n01] [2026-07-09T08:59:13.924Z] ok: [an-a01n02] [2026-07-09T08:59:13.924Z] [2026-07-09T08:59:13.924Z] TASK [check if vg already exists _raw_params=vgs anvil-test-vg [2026-07-09T08:59:13.924Z] ] **************** [2026-07-09T08:59:14.669Z] ok: [an-a01n02] [2026-07-09T08:59:14.669Z] ok: [an-a01n01] [2026-07-09T08:59:14.669Z] [2026-07-09T08:59:14.669Z] TASK [vgcheck _raw_params=end_play] ******************************************** [2026-07-09T08:59:14.669Z] skipping: [an-a01n01] [2026-07-09T08:59:14.669Z] [2026-07-09T08:59:14.669Z] TASK [create anvil-test-vg vg=anvil-test-vg, pvs=/dev/vdb] ********************* [2026-07-09T08:59:15.868Z] changed: [an-a01n02] [2026-07-09T08:59:15.868Z] changed: [an-a01n01] [2026-07-09T08:59:15.868Z] [2026-07-09T08:59:15.868Z] PLAY [deploy anvil] ************************************************************ [2026-07-09T08:59:15.868Z] [2026-07-09T08:59:15.868Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T08:59:17.957Z] ok: [an-striker02] [2026-07-09T08:59:17.957Z] ok: [an-striker01] [2026-07-09T08:59:17.957Z] [2026-07-09T08:59:17.957Z] TASK [check if installation is in progress path=/root/.anvil_deployment_in_progress] *** [2026-07-09T08:59:18.303Z] ok: [an-striker01] [2026-07-09T08:59:18.303Z] ok: [an-striker02] [2026-07-09T08:59:18.303Z] [2026-07-09T08:59:18.303Z] TASK [register installation is in progress content=, dest=/root/.anvil_deployment_in_progress, force=True, owner=root, group=root, mode=0644] *** [2026-07-09T08:59:20.033Z] changed: [an-striker02] [2026-07-09T08:59:20.033Z] changed: [an-striker01] [2026-07-09T08:59:20.033Z] [2026-07-09T08:59:20.033Z] TASK [start anvil deployment and configuration _raw_params=striker-auto-initialize-all -vv --log-secure --config /root/anvil_striker_auto.cfg [2026-07-09T08:59:20.033Z] ] *** [2026-07-09T08:59:22.375Z] changed: [an-striker01] [2026-07-09T08:59:22.375Z] changed: [an-striker02] [2026-07-09T08:59:22.375Z] [2026-07-09T08:59:22.375Z] TASK [Wait up to 20 minutes for DB to be running on both strikers _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T08:59:22.375Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T08:59:22.375Z] --username=postgres anvil -t -c "SELECT COUNT(*) FROM hosts WHERE host_type = 'striker';" [2026-07-09T08:59:22.375Z] ] *** [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-07-09T09:05:17.784Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-07-09T09:06:56.397Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (82 retries left). [2026-07-09T09:06:56.397Z] ok: [an-striker02 -> localhost] [2026-07-09T09:06:56.397Z] ok: [an-striker01 -> localhost] [2026-07-09T09:06:56.397Z] [2026-07-09T09:06:56.397Z] TASK [Wait up to 20 minutes for both striker-db-status to sync _raw_params=dbstatus="$(striker-db-status -vv --log-secure)" [2026-07-09T09:06:56.397Z] dbcon=$(echo "$dbstatus" | grep ^connections | sed -e 's#.*=##g') [2026-07-09T09:06:56.397Z] dbhosts=$(echo "$dbstatus" | grep ^host_name | wc -l) [2026-07-09T09:06:56.397Z] if [ "$dbcon" = 1 ] && [ "$dbhosts" = 2 ]; then [2026-07-09T09:06:56.397Z] echo "DB has completed sync" [2026-07-09T09:06:56.397Z] exit 0 [2026-07-09T09:06:56.397Z] fi [2026-07-09T09:06:56.397Z] if [ "$dbcon" != 2 ]; then [2026-07-09T09:06:56.397Z] echo "Detected $dbcon db connections (expected 2)" [2026-07-09T09:06:56.397Z] exit 1 [2026-07-09T09:06:56.397Z] fi [2026-07-09T09:06:56.397Z] ] *** [2026-07-09T09:07:54.492Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-07-09T09:07:54.492Z] ok: [an-striker01] [2026-07-09T09:07:54.492Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-07-09T09:07:54.492Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left). [2026-07-09T09:07:54.492Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (117 retries left). [2026-07-09T09:07:54.492Z] ok: [an-striker02] [2026-07-09T09:07:54.492Z] [2026-07-09T09:07:54.492Z] TASK [wait up to 20 minutes for nodes to join the database _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T09:07:54.492Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T09:07:54.492Z] --username=postgres anvil -t \ [2026-07-09T09:07:54.492Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::configured' ORDER BY host_name ASC;" | \ [2026-07-09T09:07:54.492Z] grep altee | \ [2026-07-09T09:07:54.492Z] grep 1$ | \ [2026-07-09T09:07:54.492Z] wc -l [2026-07-09T09:07:54.492Z] ] *** [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-07-09T09:13:34.709Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (71 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (70 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-07-09T09:16:58.977Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (69 retries left). [2026-07-09T09:16:58.977Z] ok: [an-striker02 -> localhost] [2026-07-09T09:16:58.977Z] ok: [an-striker01 -> localhost] [2026-07-09T09:16:58.977Z] [2026-07-09T09:16:58.977Z] TASK [wait up to 20 minutes for anvil nodes to form _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T09:16:58.977Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T09:16:58.977Z] --username=postgres anvil -t -c "SELECT a.host_name AS node1, b.host_name AS node2, d.anvil_name FROM hosts a, hosts b, anvils d WHERE a.host_uuid = d.anvil_node1_host_uuid AND b.host_uuid = d.anvil_node2_host_uuid;" | \ [2026-07-09T09:16:58.977Z] grep altee | \ [2026-07-09T09:16:58.977Z] wc -l [2026-07-09T09:16:58.977Z] ] *** [2026-07-09T09:16:58.977Z] ok: [an-striker01 -> localhost] [2026-07-09T09:16:58.977Z] ok: [an-striker02 -> localhost] [2026-07-09T09:16:58.977Z] [2026-07-09T09:16:58.977Z] TASK [wait up to 60 minutes for pacemaker to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T09:16:58.977Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T09:16:58.977Z] --username=postgres anvil -t \ [2026-07-09T09:16:58.977Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::pacemaker::configured' ORDER BY host_name ASC;" | \ [2026-07-09T09:16:58.977Z] grep altee | \ [2026-07-09T09:16:58.977Z] grep 1$ | \ [2026-07-09T09:16:58.977Z] wc -l [2026-07-09T09:16:58.977Z] ] *** [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-07-09T09:18:37.517Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-07-09T09:18:37.517Z] ok: [an-striker02 -> localhost] [2026-07-09T09:18:37.517Z] ok: [an-striker01 -> localhost] [2026-07-09T09:18:37.517Z] [2026-07-09T09:18:37.517Z] TASK [wait up to 20 minutes for drbd to be configured _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T09:18:37.517Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T09:18:37.517Z] --username=postgres anvil -t \ [2026-07-09T09:18:37.517Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::drbd::configured' ORDER BY host_name ASC;" | \ [2026-07-09T09:18:37.517Z] grep altee | \ [2026-07-09T09:18:37.517Z] grep 1$ | \ [2026-07-09T09:18:37.517Z] wc -l [2026-07-09T09:18:37.517Z] ] *** [2026-07-09T09:18:43.573Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-07-09T09:18:43.573Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-07-09T09:18:43.573Z] ok: [an-striker01 -> localhost] [2026-07-09T09:18:43.573Z] ok: [an-striker02 -> localhost] [2026-07-09T09:18:43.573Z] [2026-07-09T09:18:43.573Z] TASK [wait up to 20 minutes for any other configuration job to complete _raw_params=export PGPASSWORD={{ pgpasswd }} && \ [2026-07-09T09:18:43.573Z] psql --host="{{ inventory_hostname }}" \ [2026-07-09T09:18:43.573Z] --username=postgres anvil -t \ [2026-07-09T09:18:43.573Z] -c "SELECT DISTINCT a.host_name, b.variable_value FROM hosts a, variables b WHERE a.host_uuid = b.variable_source_uuid AND b.variable_name = 'system::all::configured' ORDER BY host_name ASC;" | \ [2026-07-09T09:18:43.573Z] grep altee | \ [2026-07-09T09:18:43.573Z] grep 1$ | \ [2026-07-09T09:18:43.573Z] wc -l [2026-07-09T09:18:43.573Z] ] *** [2026-07-09T09:18:54.602Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-07-09T09:18:54.602Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-07-09T09:18:54.602Z] ok: [an-striker01 -> localhost] [2026-07-09T09:18:54.602Z] ok: [an-striker02 -> localhost] [2026-07-09T09:18:54.602Z] [2026-07-09T09:18:54.602Z] PLAY [check network status after deployment] *********************************** [2026-07-09T09:18:54.602Z] [2026-07-09T09:18:54.602Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:18:56.939Z] ok: [an-a01n02] [2026-07-09T09:18:56.939Z] ok: [an-a01n01] [2026-07-09T09:18:56.939Z] ok: [an-striker01] [2026-07-09T09:18:56.939Z] ok: [an-striker02] [2026-07-09T09:18:56.939Z] [2026-07-09T09:18:56.939Z] TASK [check network manager autoconnect is set to yes _raw_params=device="$(nmcli device show {{ item }} | grep "GENERAL.CONNECTION:" | awk '{print $NF}')" [2026-07-09T09:18:56.939Z] if [ "$device" = "" ]; then [2026-07-09T09:18:56.939Z] echo "ERROR: unable to determine device for iface {{ item }}" [2026-07-09T09:18:56.939Z] exit 1 [2026-07-09T09:18:56.939Z] fi [2026-07-09T09:18:56.939Z] if [ "$(nmcli connection show $device | grep 'autoconnect:' | awk '{print $NF}' | sort -u)" != "yes" ]; then [2026-07-09T09:18:56.939Z] echo "ERROR: {{ item }} / $device is not set to autoconnecct" [2026-07-09T09:18:56.939Z] exit 1 [2026-07-09T09:18:56.939Z] fi [2026-07-09T09:18:56.939Z] exit 0 [2026-07-09T09:18:56.939Z] ] *** [2026-07-09T09:19:07.967Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-07-09T09:19:07.967Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-07-09T09:19:07.967Z] skipping: [an-a01n02] => (item=lo) [2026-07-09T09:19:07.967Z] changed: [an-striker02] => (item=ifn1_link2) [2026-07-09T09:19:07.967Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-07-09T09:19:07.967Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-07-09T09:19:07.967Z] changed: [an-striker01] => (item=ifn1_link2) [2026-07-09T09:19:07.967Z] changed: [an-striker02] => (item=bcn1_link1) [2026-07-09T09:19:07.967Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-07-09T09:19:07.967Z] changed: [an-a01n02] => (item=sn1_link2) [2026-07-09T09:19:07.967Z] changed: [an-a01n01] => (item=sn1_link2) [2026-07-09T09:19:07.967Z] skipping: [an-a01n01] => (item=lo) [2026-07-09T09:19:07.967Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-07-09T09:19:07.967Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-07-09T09:19:07.967Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-07-09T09:19:07.967Z] changed: [an-a01n01] => (item=sn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=sn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-07-09T09:19:07.968Z] changed: [an-striker01] => (item=bcn1_link1) [2026-07-09T09:19:07.968Z] skipping: [an-striker01] => (item=lo) [2026-07-09T09:19:07.968Z] changed: [an-striker02] => (item=bcn1_link2) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-07-09T09:19:07.968Z] changed: [an-striker02] => (item=ifn1_link1) [2026-07-09T09:19:07.968Z] skipping: [an-striker02] => (item=lo) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-07-09T09:19:07.968Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-07-09T09:19:07.968Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-07-09T09:19:07.968Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-07-09T09:19:07.968Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-07-09T09:19:07.968Z] changed: [an-striker01] => (item=ifn1_link1) [2026-07-09T09:19:07.968Z] changed: [an-striker01] => (item=bcn1_link2) [2026-07-09T09:19:07.968Z] [2026-07-09T09:19:07.968Z] PLAY [check pacemaker status after deployment] ********************************* [2026-07-09T09:19:07.968Z] [2026-07-09T09:19:07.968Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:19:08.313Z] ok: [an-a01n01] [2026-07-09T09:19:08.314Z] ok: [an-a01n02] [2026-07-09T09:19:08.314Z] [2026-07-09T09:19:08.314Z] TASK [check cluster is in good state _raw_params=../common/cluster_good.yml] *** [2026-07-09T09:19:08.314Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T09:19:08.314Z] [2026-07-09T09:19:08.314Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T09:19:08.314Z] ] *** [2026-07-09T09:19:12.233Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T09:19:12.233Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T09:19:12.233Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T09:19:12.233Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T09:19:12.233Z] [2026-07-09T09:19:12.233Z] PLAY [check drbd kmod] ********************************************************* [2026-07-09T09:19:12.233Z] [2026-07-09T09:19:12.233Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:19:13.956Z] ok: [an-a01n01] [2026-07-09T09:19:13.956Z] ok: [an-a01n02] [2026-07-09T09:19:13.956Z] [2026-07-09T09:19:13.956Z] TASK [check if drbd kernel module is already loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-07-09T09:19:13.956Z] ] *** [2026-07-09T09:19:14.698Z] ok: [an-a01n01] [2026-07-09T09:19:14.698Z] ok: [an-a01n02] [2026-07-09T09:19:14.698Z] [2026-07-09T09:19:14.698Z] TASK [load drbd kernel module _raw_params=modprobe drbd [2026-07-09T09:19:14.698Z] ] *********************** [2026-07-09T09:19:14.698Z] skipping: [an-a01n01] [2026-07-09T09:19:14.698Z] skipping: [an-a01n02] [2026-07-09T09:19:14.698Z] [2026-07-09T09:19:14.698Z] TASK [check if drbd kernel module is loaded _raw_params=lsmod 2>/dev/null| grep -q drbd [2026-07-09T09:19:14.698Z] ] *** [2026-07-09T09:19:14.698Z] skipping: [an-a01n01] [2026-07-09T09:19:14.698Z] skipping: [an-a01n02] [2026-07-09T09:19:14.698Z] [2026-07-09T09:19:14.698Z] PLAY [Copy iso images to striker01] ******************************************** [2026-07-09T09:19:14.698Z] [2026-07-09T09:19:14.698Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:19:17.158Z] ok: [an-striker01] [2026-07-09T09:19:17.158Z] [2026-07-09T09:19:17.158Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-07-09T09:19:18.470Z] ok: [an-striker01] [2026-07-09T09:19:18.470Z] [2026-07-09T09:19:18.470Z] TASK [Create snapshots dir path=/tmp/images, state=directory, mode=0o700] ****** [2026-07-09T09:19:19.597Z] changed: [an-striker01] [2026-07-09T09:19:19.597Z] [2026-07-09T09:19:19.597Z] TASK [Copy C9S kickstart images src={{ deploydir }}/snapshots/{{ item }}, dest=/tmp/images/{{ item }}, owner=root, group=root, mode=0644] *** [2026-07-09T09:19:26.553Z] changed: [an-striker01] => (item=deploy1.iso) [2026-07-09T09:19:26.553Z] changed: [an-striker01] => (item=deploy2.iso) [2026-07-09T09:19:26.553Z] changed: [an-striker01] => (item=deploy3.iso) [2026-07-09T09:19:26.553Z] changed: [an-striker01] => (item=deploy4.iso) [2026-07-09T09:19:26.553Z] changed: [an-striker01] => (item=deploy5.iso) [2026-07-09T09:19:26.553Z] [2026-07-09T09:19:26.553Z] TASK [Move install isos to anvil upload dir _raw_params=mv /tmp/images/*.iso /mnt/shared/incoming/ [2026-07-09T09:19:26.553Z] ] *** [2026-07-09T09:19:27.294Z] changed: [an-striker01] [2026-07-09T09:19:27.294Z] [2026-07-09T09:19:27.294Z] TASK [Wipe old snapshots dir path=/tmp/images, state=absent] ******************* [2026-07-09T09:19:28.490Z] changed: [an-striker01] [2026-07-09T09:19:28.490Z] [2026-07-09T09:19:28.490Z] PLAY [Check iso images propagation] ******************************************** [2026-07-09T09:19:28.490Z] [2026-07-09T09:19:28.490Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:19:30.214Z] ok: [an-striker01] [2026-07-09T09:19:30.214Z] ok: [an-a01n02] [2026-07-09T09:19:30.214Z] ok: [an-striker02] [2026-07-09T09:19:30.214Z] ok: [an-a01n01] [2026-07-09T09:19:30.214Z] [2026-07-09T09:19:30.214Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/files/*.iso | wc -l)" != 6 ]; then [2026-07-09T09:19:30.214Z] exit 1 [2026-07-09T09:19:30.214Z] else [2026-07-09T09:19:30.214Z] exit 0 [2026-07-09T09:19:30.214Z] fi [2026-07-09T09:19:30.214Z] ] *** [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (173 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (172 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-07-09T09:21:23.846Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (171 retries left). [2026-07-09T09:21:23.846Z] ok: [an-a01n02] [2026-07-09T09:21:23.846Z] ok: [an-a01n01] [2026-07-09T09:21:23.846Z] ok: [an-striker01] [2026-07-09T09:21:23.846Z] ok: [an-striker02] [2026-07-09T09:21:23.846Z] [2026-07-09T09:21:23.846Z] PLAY [example tests] *********************************************************** [2026-07-09T09:21:23.846Z] [2026-07-09T09:21:23.846Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:21:25.042Z] ok: [an-striker02] [2026-07-09T09:21:25.042Z] ok: [an-a01n02] [2026-07-09T09:21:25.042Z] ok: [an-a01n01] [2026-07-09T09:21:25.042Z] ok: [an-striker01] [2026-07-09T09:21:25.042Z] [2026-07-09T09:21:25.042Z] TASK [this test should be executed only on strikers _raw_params=/usr/bin/true [2026-07-09T09:21:25.042Z] ] *** [2026-07-09T09:21:26.239Z] skipping: [an-a01n01] [2026-07-09T09:21:26.239Z] skipping: [an-a01n02] [2026-07-09T09:21:26.239Z] ok: [an-striker01] [2026-07-09T09:21:26.239Z] ok: [an-striker02] [2026-07-09T09:21:26.239Z] [2026-07-09T09:21:26.239Z] TASK [this test should be executed only on non strikers _raw_params=/usr/bin/true [2026-07-09T09:21:26.239Z] ] *** [2026-07-09T09:21:26.239Z] skipping: [an-striker01] [2026-07-09T09:21:26.239Z] skipping: [an-striker02] [2026-07-09T09:21:26.584Z] ok: [an-a01n01] [2026-07-09T09:21:26.584Z] ok: [an-a01n02] [2026-07-09T09:21:26.584Z] [2026-07-09T09:21:26.584Z] TASK [inject failure _raw_params=/usr/bin/false [2026-07-09T09:21:26.584Z] ] ******************************* [2026-07-09T09:21:26.930Z] skipping: [an-striker01] [2026-07-09T09:21:26.930Z] skipping: [an-striker02] [2026-07-09T09:21:26.930Z] skipping: [an-a01n01] [2026-07-09T09:21:26.930Z] skipping: [an-a01n02] [2026-07-09T09:21:26.930Z] [2026-07-09T09:21:26.930Z] PLAY [Record anvil available resources from striker01] ************************* [2026-07-09T09:21:26.930Z] [2026-07-09T09:21:26.930Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:21:28.652Z] ok: [an-striker01] [2026-07-09T09:21:28.652Z] [2026-07-09T09:21:28.652Z] TASK [Record anvil available resources _raw_params=anvil-report-usage > /root/anvil_available_resources.start [2026-07-09T09:21:28.652Z] ] *** [2026-07-09T09:21:32.545Z] ok: [an-striker01] [2026-07-09T09:21:32.545Z] [2026-07-09T09:21:32.545Z] PLAY [check selinux status] **************************************************** [2026-07-09T09:21:32.545Z] [2026-07-09T09:21:32.545Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:21:34.270Z] ok: [an-a01n02] [2026-07-09T09:21:34.270Z] ok: [an-a01n01] [2026-07-09T09:21:34.270Z] ok: [an-striker01] [2026-07-09T09:21:34.270Z] ok: [an-striker02] [2026-07-09T09:21:34.270Z] [2026-07-09T09:21:34.270Z] TASK [check selinux status permissive (8) var=ansible_selinux] ***************** [2026-07-09T09:21:34.270Z] skipping: [an-striker01] [2026-07-09T09:21:34.270Z] skipping: [an-striker02] [2026-07-09T09:21:34.270Z] skipping: [an-a01n01] [2026-07-09T09:21:34.270Z] skipping: [an-a01n02] [2026-07-09T09:21:34.270Z] [2026-07-09T09:21:34.270Z] TASK [check selinux status permissive (9) var=ansible_selinux] ***************** [2026-07-09T09:21:34.617Z] ok: [an-striker01] => { [2026-07-09T09:21:34.617Z] "ansible_selinux": { [2026-07-09T09:21:34.617Z] "config_mode": "enforcing", [2026-07-09T09:21:34.617Z] "mode": "enforcing", [2026-07-09T09:21:34.617Z] "policyvers": 33, [2026-07-09T09:21:34.617Z] "status": "enabled", [2026-07-09T09:21:34.617Z] "type": "targeted" [2026-07-09T09:21:34.617Z] }, [2026-07-09T09:21:34.617Z] "failed_when_result": false [2026-07-09T09:21:34.617Z] } [2026-07-09T09:21:34.617Z] ok: [an-striker02] => { [2026-07-09T09:21:34.617Z] "ansible_selinux": { [2026-07-09T09:21:34.617Z] "config_mode": "enforcing", [2026-07-09T09:21:34.617Z] "mode": "enforcing", [2026-07-09T09:21:34.617Z] "policyvers": 33, [2026-07-09T09:21:34.617Z] "status": "enabled", [2026-07-09T09:21:34.617Z] "type": "targeted" [2026-07-09T09:21:34.617Z] }, [2026-07-09T09:21:34.617Z] "failed_when_result": false [2026-07-09T09:21:34.617Z] } [2026-07-09T09:21:34.617Z] ok: [an-a01n01] => { [2026-07-09T09:21:34.617Z] "ansible_selinux": { [2026-07-09T09:21:34.617Z] "config_mode": "enforcing", [2026-07-09T09:21:34.617Z] "mode": "enforcing", [2026-07-09T09:21:34.617Z] "policyvers": 33, [2026-07-09T09:21:34.617Z] "status": "enabled", [2026-07-09T09:21:34.617Z] "type": "targeted" [2026-07-09T09:21:34.617Z] }, [2026-07-09T09:21:34.617Z] "failed_when_result": false [2026-07-09T09:21:34.617Z] } [2026-07-09T09:21:34.617Z] ok: [an-a01n02] => { [2026-07-09T09:21:34.617Z] "ansible_selinux": { [2026-07-09T09:21:34.617Z] "config_mode": "enforcing", [2026-07-09T09:21:34.617Z] "mode": "enforcing", [2026-07-09T09:21:34.617Z] "policyvers": 33, [2026-07-09T09:21:34.617Z] "status": "enabled", [2026-07-09T09:21:34.617Z] "type": "targeted" [2026-07-09T09:21:34.617Z] }, [2026-07-09T09:21:34.617Z] "failed_when_result": false [2026-07-09T09:21:34.617Z] } [2026-07-09T09:21:34.617Z] [2026-07-09T09:21:34.617Z] PLAY [Create random file on striker01] ***************************************** [2026-07-09T09:21:34.617Z] [2026-07-09T09:21:34.617Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:21:36.953Z] ok: [an-striker01] [2026-07-09T09:21:36.953Z] [2026-07-09T09:21:36.953Z] TASK [Create archive file content=foobarbaz, dest=/mnt/shared/archives/something.test, force=True, owner=root, group=root, mode=0644] *** [2026-07-09T09:21:38.678Z] changed: [an-striker01] [2026-07-09T09:21:38.678Z] [2026-07-09T09:21:38.678Z] PLAY [Check archive file propagation] ****************************************** [2026-07-09T09:21:38.678Z] [2026-07-09T09:21:38.678Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:21:40.402Z] ok: [an-a01n02] [2026-07-09T09:21:40.402Z] ok: [an-a01n01] [2026-07-09T09:21:40.402Z] ok: [an-striker01] [2026-07-09T09:21:40.402Z] ok: [an-striker02] [2026-07-09T09:21:40.402Z] [2026-07-09T09:21:40.402Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes _raw_params=if [ "$(ls -1 /mnt/shared/archives/something.test | wc -l)" != 1 ]; then [2026-07-09T09:21:40.402Z] exit 1 [2026-07-09T09:21:40.402Z] else [2026-07-09T09:21:40.402Z] exit 0 [2026-07-09T09:21:40.402Z] fi [2026-07-09T09:21:40.402Z] ] *** [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-07-09T09:23:03.857Z] ok: [an-striker01] [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-07-09T09:23:03.857Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-07-09T09:23:03.857Z] ok: [an-striker02] [2026-07-09T09:23:03.857Z] ok: [an-a01n01] [2026-07-09T09:23:03.857Z] ok: [an-a01n02] [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] PLAY [test fencing] ************************************************************ [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:23:03.857Z] ok: [an-a01n01] [2026-07-09T09:23:03.857Z] ok: [an-a01n02] [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T09:23:03.857Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T09:23:03.857Z] ] *** [2026-07-09T09:23:03.857Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T09:23:03.857Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T09:23:03.857Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T09:23:03.857Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [kill node "an-a01n01" _raw_params=../../common/kill_node.yml] ************ [2026-07-09T09:23:03.857Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T09:23:03.857Z] ] ********** [2026-07-09T09:23:03.857Z] skipping: [an-a01n02] [2026-07-09T09:23:03.857Z] changed: [an-a01n01] [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T09:23:03.857Z] skipping: [an-a01n02] [2026-07-09T09:23:03.857Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-07-09T09:23:03.857Z] [2026-07-09T09:23:03.857Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T09:26:12.988Z] ok: [an-a01n01 -> localhost] [2026-07-09T09:26:12.988Z] [2026-07-09T09:26:12.988Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T09:26:12.988Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T09:26:12.988Z] [2026-07-09T09:26:12.988Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T09:26:12.988Z] ] *** [2026-07-09T09:26:16.882Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T09:26:16.882Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T09:26:16.882Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T09:26:16.882Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T09:26:16.882Z] [2026-07-09T09:26:16.882Z] TASK [kill node "an-a01n02" _raw_params=../../common/kill_node.yml] ************ [2026-07-09T09:26:16.882Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T09:26:16.882Z] [2026-07-09T09:26:16.882Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T09:26:16.882Z] ] ********** [2026-07-09T09:26:17.625Z] skipping: [an-a01n01] [2026-07-09T09:26:17.625Z] changed: [an-a01n02] [2026-07-09T09:26:17.625Z] [2026-07-09T09:26:17.625Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T09:26:17.625Z] skipping: [an-a01n01] [2026-07-09T09:26:17.625Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-07-09T09:26:17.625Z] [2026-07-09T09:26:17.625Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T09:29:41.850Z] ok: [an-a01n02 -> localhost] [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T09:29:41.850Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T09:29:41.850Z] ] *** [2026-07-09T09:29:41.850Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T09:29:41.850Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T09:29:41.850Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T09:29:41.850Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] PLAY [advanced server tests] *************************************************** [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T09:29:41.850Z] ok: [an-a01n01] [2026-07-09T09:29:41.850Z] ok: [an-a01n02] [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T09:29:41.850Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T09:29:41.850Z] ] *** [2026-07-09T09:29:41.850Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T09:29:41.850Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T09:29:41.850Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T09:29:41.850Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-07-09T09:29:41.850Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-07-09T09:29:41.850Z] [2026-07-09T09:29:41.850Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-07-09T09:29:41.850Z] if [ -z "$storage" ]; then [2026-07-09T09:29:41.850Z] echo "Unable to detect shared storage" [2026-07-09T09:29:41.850Z] exit 1 [2026-07-09T09:29:41.850Z] fi [2026-07-09T09:29:41.850Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T09:29:41.850Z] ] *** [2026-07-09T09:29:43.575Z] changed: [an-a01n01] [2026-07-09T09:29:43.575Z] changed: [an-a01n02] [2026-07-09T09:29:43.575Z] [2026-07-09T09:29:43.575Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-07-09T09:29:43.575Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-07-09T09:29:43.575Z] [2026-07-09T09:29:43.575Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-07-09T09:29:43.575Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-07-09T09:29:43.575Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-07-09T09:29:43.575Z] echo "Unable to detect all deploy iso" [2026-07-09T09:29:43.575Z] echo "Detected isos" [2026-07-09T09:29:43.575Z] echo "$isos" [2026-07-09T09:29:43.575Z] exit 1 [2026-07-09T09:29:43.575Z] fi [2026-07-09T09:29:43.575Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-07-09T09:29:43.575Z] if [ -z "$installiso" ]; then [2026-07-09T09:29:43.575Z] echo "Unable to detect install iso" [2026-07-09T09:29:43.575Z] echo "Detected isos" [2026-07-09T09:29:43.575Z] echo "$isos" [2026-07-09T09:29:43.575Z] exit 1 [2026-07-09T09:29:43.575Z] fi [2026-07-09T09:29:43.575Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T09:29:43.575Z] ] *** [2026-07-09T09:29:49.630Z] changed: [an-a01n02] [2026-07-09T09:29:49.630Z] changed: [an-a01n01] [2026-07-09T09:29:49.630Z] [2026-07-09T09:29:49.630Z] TASK [deploy servers _raw_params=../../common/deploy_server.yml] *************** [2026-07-09T09:29:49.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T09:29:49.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T09:29:49.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T09:29:49.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T09:29:49.630Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T09:29:49.630Z] [2026-07-09T09:29:49.630Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T09:29:49.630Z] ] *** [2026-07-09T09:29:50.370Z] skipping: [an-a01n02] [2026-07-09T09:29:50.370Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.942200", "end": "2026-07-09 09:29:50.118469", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 09:29:49.176269", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T09:29:50.370Z] ...ignoring [2026-07-09T09:29:50.370Z] [2026-07-09T09:29:50.370Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T09:29:50.370Z] --ci-test \ [2026-07-09T09:29:50.370Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T09:29:50.370Z] --os almalinux9 \ [2026-07-09T09:29:50.370Z] --cpu 2 --ram 4G \ [2026-07-09T09:29:50.370Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T09:29:50.370Z] --install-media {{ installisouuid }} \ [2026-07-09T09:29:50.370Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T09:29:50.370Z] ] *** [2026-07-09T09:29:59.440Z] skipping: [an-a01n02] [2026-07-09T09:29:59.440Z] changed: [an-a01n01] [2026-07-09T09:29:59.440Z] [2026-07-09T09:29:59.440Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T09:29:59.440Z] skipping: [an-a01n01] [2026-07-09T09:29:59.440Z] skipping: [an-a01n02] [2026-07-09T09:29:59.440Z] [2026-07-09T09:29:59.440Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T09:29:59.440Z] skipping: [an-a01n01] [2026-07-09T09:29:59.440Z] skipping: [an-a01n02] [2026-07-09T09:29:59.440Z] [2026-07-09T09:29:59.440Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T09:29:59.440Z] skipping: [an-a01n01] [2026-07-09T09:29:59.440Z] skipping: [an-a01n02] [2026-07-09T09:29:59.440Z] [2026-07-09T09:29:59.440Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T09:29:59.440Z] skipping: [an-a01n01] [2026-07-09T09:29:59.440Z] skipping: [an-a01n02] [2026-07-09T09:29:59.440Z] [2026-07-09T09:29:59.440Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T09:29:59.440Z] ] *** [2026-07-09T09:30:00.640Z] skipping: [an-a01n02] [2026-07-09T09:30:00.640Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.940790", "end": "2026-07-09 09:30:00.452139", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 09:29:59.511349", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T09:30:00.640Z] ...ignoring [2026-07-09T09:30:00.640Z] [2026-07-09T09:30:00.640Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T09:30:00.640Z] --ci-test \ [2026-07-09T09:30:00.640Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T09:30:00.640Z] --os almalinux9 \ [2026-07-09T09:30:00.640Z] --cpu 2 --ram 4G \ [2026-07-09T09:30:00.640Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T09:30:00.640Z] --install-media {{ installisouuid }} \ [2026-07-09T09:30:00.640Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T09:30:00.640Z] ] *** [2026-07-09T09:30:09.714Z] skipping: [an-a01n02] [2026-07-09T09:30:09.714Z] changed: [an-a01n01] [2026-07-09T09:30:09.714Z] [2026-07-09T09:30:09.714Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T09:30:09.714Z] skipping: [an-a01n01] [2026-07-09T09:30:09.714Z] skipping: [an-a01n02] [2026-07-09T09:30:09.714Z] [2026-07-09T09:30:09.714Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T09:30:09.714Z] skipping: [an-a01n01] [2026-07-09T09:30:09.714Z] skipping: [an-a01n02] [2026-07-09T09:30:09.714Z] [2026-07-09T09:30:09.714Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T09:30:09.714Z] skipping: [an-a01n01] [2026-07-09T09:30:09.714Z] skipping: [an-a01n02] [2026-07-09T09:30:09.714Z] [2026-07-09T09:30:09.714Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T09:30:10.060Z] skipping: [an-a01n01] [2026-07-09T09:30:10.060Z] skipping: [an-a01n02] [2026-07-09T09:30:10.060Z] [2026-07-09T09:30:10.060Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T09:30:10.060Z] ] *** [2026-07-09T09:30:11.785Z] skipping: [an-a01n02] [2026-07-09T09:30:11.785Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:01.016826", "end": "2026-07-09 09:30:11.269268", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 09:30:10.252442", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T09:30:11.785Z] ...ignoring [2026-07-09T09:30:11.785Z] [2026-07-09T09:30:11.785Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T09:30:11.785Z] --ci-test \ [2026-07-09T09:30:11.785Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T09:30:11.785Z] --os almalinux9 \ [2026-07-09T09:30:11.785Z] --cpu 2 --ram 4G \ [2026-07-09T09:30:11.785Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T09:30:11.785Z] --install-media {{ installisouuid }} \ [2026-07-09T09:30:11.785Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T09:30:11.785Z] ] *** [2026-07-09T09:30:20.858Z] skipping: [an-a01n02] [2026-07-09T09:30:20.858Z] changed: [an-a01n01] [2026-07-09T09:30:20.858Z] [2026-07-09T09:30:20.858Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T09:30:20.858Z] skipping: [an-a01n01] [2026-07-09T09:30:20.858Z] skipping: [an-a01n02] [2026-07-09T09:30:20.858Z] [2026-07-09T09:30:20.858Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T09:30:20.858Z] skipping: [an-a01n01] [2026-07-09T09:30:20.858Z] skipping: [an-a01n02] [2026-07-09T09:30:20.858Z] [2026-07-09T09:30:20.858Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T09:30:20.858Z] skipping: [an-a01n01] [2026-07-09T09:30:20.858Z] skipping: [an-a01n02] [2026-07-09T09:30:20.858Z] [2026-07-09T09:30:20.858Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T09:30:20.858Z] skipping: [an-a01n01] [2026-07-09T09:30:20.858Z] skipping: [an-a01n02] [2026-07-09T09:30:20.858Z] [2026-07-09T09:30:20.858Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T09:30:20.858Z] ] *** [2026-07-09T09:30:22.055Z] skipping: [an-a01n02] [2026-07-09T09:30:22.055Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.930716", "end": "2026-07-09 09:30:21.890655", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 09:30:20.959939", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T09:30:22.055Z] ...ignoring [2026-07-09T09:30:22.055Z] [2026-07-09T09:30:22.055Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T09:30:22.055Z] --ci-test \ [2026-07-09T09:30:22.055Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T09:30:22.055Z] --os almalinux9 \ [2026-07-09T09:30:22.055Z] --cpu 2 --ram 4G \ [2026-07-09T09:30:22.055Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T09:30:22.055Z] --install-media {{ installisouuid }} \ [2026-07-09T09:30:22.055Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T09:30:22.055Z] ] *** [2026-07-09T09:30:31.126Z] skipping: [an-a01n02] [2026-07-09T09:30:31.126Z] changed: [an-a01n01] [2026-07-09T09:30:31.126Z] [2026-07-09T09:30:31.126Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T09:30:31.126Z] skipping: [an-a01n01] [2026-07-09T09:30:31.126Z] skipping: [an-a01n02] [2026-07-09T09:30:31.126Z] [2026-07-09T09:30:31.126Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T09:30:31.126Z] skipping: [an-a01n01] [2026-07-09T09:30:31.126Z] skipping: [an-a01n02] [2026-07-09T09:30:31.126Z] [2026-07-09T09:30:31.126Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T09:30:31.126Z] skipping: [an-a01n01] [2026-07-09T09:30:31.126Z] skipping: [an-a01n02] [2026-07-09T09:30:31.126Z] [2026-07-09T09:30:31.126Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T09:30:31.472Z] skipping: [an-a01n01] [2026-07-09T09:30:31.472Z] skipping: [an-a01n02] [2026-07-09T09:30:31.472Z] [2026-07-09T09:30:31.472Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T09:30:31.472Z] ] *** [2026-07-09T09:30:33.197Z] skipping: [an-a01n02] [2026-07-09T09:30:33.197Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.917444", "end": "2026-07-09 09:30:32.616834", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 09:30:31.699390", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T09:30:33.197Z] ...ignoring [2026-07-09T09:30:33.197Z] [2026-07-09T09:30:33.197Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T09:30:33.197Z] --ci-test \ [2026-07-09T09:30:33.197Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T09:30:33.197Z] --os almalinux9 \ [2026-07-09T09:30:33.197Z] --cpu 2 --ram 4G \ [2026-07-09T09:30:33.197Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T09:30:33.197Z] --install-media {{ installisouuid }} \ [2026-07-09T09:30:33.197Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T09:30:33.197Z] ] *** [2026-07-09T09:30:42.270Z] skipping: [an-a01n02] [2026-07-09T09:30:42.270Z] changed: [an-a01n01] [2026-07-09T09:30:42.270Z] [2026-07-09T09:30:42.270Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T09:30:42.270Z] skipping: [an-a01n01] [2026-07-09T09:30:42.270Z] skipping: [an-a01n02] [2026-07-09T09:30:42.270Z] [2026-07-09T09:30:42.270Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T09:30:42.270Z] skipping: [an-a01n01] [2026-07-09T09:30:42.270Z] skipping: [an-a01n02] [2026-07-09T09:30:42.270Z] [2026-07-09T09:30:42.270Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T09:30:42.617Z] skipping: [an-a01n01] [2026-07-09T09:30:42.617Z] skipping: [an-a01n02] [2026-07-09T09:30:42.617Z] [2026-07-09T09:30:42.617Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T09:30:42.617Z] skipping: [an-a01n01] [2026-07-09T09:30:42.617Z] skipping: [an-a01n02] [2026-07-09T09:30:42.617Z] [2026-07-09T09:30:42.617Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-07-09T09:30:42.617Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T09:30:42.617Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T09:30:42.617Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T09:30:42.617Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T09:30:42.617Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T09:30:42.617Z] [2026-07-09T09:30:42.617Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T09:30:42.617Z] ] *** [2026-07-09T09:31:10.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-07-09T09:31:10.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-07-09T09:31:10.640Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-07-09T09:31:10.640Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-07-09T09:31:10.640Z] ok: [an-a01n02] [2026-07-09T09:31:10.640Z] ok: [an-a01n01] [2026-07-09T09:31:10.640Z] [2026-07-09T09:31:10.640Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T09:31:10.640Z] ] *** [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left). [2026-07-09T09:33:34.467Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left). [2026-07-09T09:33:34.467Z] ok: [an-a01n02] [2026-07-09T09:33:34.467Z] ok: [an-a01n01] [2026-07-09T09:33:34.467Z] [2026-07-09T09:33:34.467Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T09:33:34.467Z] ] *** [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-07-09T09:36:13.424Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-07-09T09:36:13.424Z] ok: [an-a01n01] [2026-07-09T09:36:13.424Z] ok: [an-a01n02] [2026-07-09T09:36:13.424Z] [2026-07-09T09:36:13.424Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T09:36:13.424Z] ] *** [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-07-09T09:38:37.257Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-07-09T09:38:37.257Z] ok: [an-a01n02] [2026-07-09T09:38:37.257Z] ok: [an-a01n01] [2026-07-09T09:38:37.257Z] [2026-07-09T09:38:37.257Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T09:38:37.257Z] ] *** [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-07-09T09:41:32.038Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-07-09T09:41:32.038Z] ok: [an-a01n02] [2026-07-09T09:41:32.038Z] ok: [an-a01n01] [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-07-09T09:41:32.038Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=1) [2026-07-09T09:41:32.038Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=2) [2026-07-09T09:41:32.038Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=3) [2026-07-09T09:41:32.038Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=4) [2026-07-09T09:41:32.038Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=5) [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T09:41:32.038Z] ] *** [2026-07-09T09:41:32.038Z] ok: [an-a01n02] [2026-07-09T09:41:32.038Z] ok: [an-a01n01] [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T09:41:32.038Z] ] *** [2026-07-09T09:41:32.038Z] ok: [an-a01n01] [2026-07-09T09:41:32.038Z] ok: [an-a01n02] [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T09:41:32.038Z] ] *** [2026-07-09T09:41:32.038Z] ok: [an-a01n02] [2026-07-09T09:41:32.038Z] ok: [an-a01n01] [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T09:41:32.038Z] ] *** [2026-07-09T09:41:32.038Z] ok: [an-a01n02] [2026-07-09T09:41:32.038Z] ok: [an-a01n01] [2026-07-09T09:41:32.038Z] [2026-07-09T09:41:32.038Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T09:41:32.038Z] ] *** [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (357 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (356 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (355 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (354 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (353 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (352 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (351 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (351 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (350 retries left). [2026-07-09T09:43:40.779Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (350 retries left). [2026-07-09T09:43:40.779Z] ok: [an-a01n02] [2026-07-09T09:43:40.779Z] ok: [an-a01n01] [2026-07-09T09:43:40.779Z] [2026-07-09T09:43:40.779Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-07-09T09:43:40.779Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T09:43:40.779Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T09:43:40.779Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T09:43:40.779Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T09:43:40.779Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T09:43:40.779Z] [2026-07-09T09:43:40.779Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T09:43:40.779Z] ] *** [2026-07-09T09:43:40.779Z] ok: [an-a01n02] [2026-07-09T09:43:40.779Z] ok: [an-a01n01] [2026-07-09T09:43:40.779Z] [2026-07-09T09:43:40.779Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T09:43:40.779Z] ] *** [2026-07-09T09:43:40.779Z] ok: [an-a01n02] [2026-07-09T09:43:40.779Z] ok: [an-a01n01] [2026-07-09T09:43:40.779Z] [2026-07-09T09:43:40.779Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T09:43:40.779Z] ] *** [2026-07-09T09:43:40.779Z] ok: [an-a01n02] [2026-07-09T09:43:40.779Z] ok: [an-a01n01] [2026-07-09T09:43:40.779Z] [2026-07-09T09:43:40.779Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T09:43:40.779Z] ] *** [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-07-09T09:46:04.637Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-07-09T09:46:04.637Z] ok: [an-a01n02] [2026-07-09T09:46:04.637Z] ok: [an-a01n01] [2026-07-09T09:46:04.637Z] [2026-07-09T09:46:04.637Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T09:46:04.637Z] ] *** [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (349 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (348 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (347 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (346 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (345 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (344 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (343 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (342 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (342 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (341 retries left). [2026-07-09T09:50:44.355Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (341 retries left). [2026-07-09T09:50:44.355Z] ok: [an-a01n01] [2026-07-09T09:50:44.355Z] ok: [an-a01n02] [2026-07-09T09:50:44.355Z] [2026-07-09T09:50:44.355Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-07-09T09:50:44.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T09:50:44.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T09:50:44.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T09:50:44.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T09:50:44.355Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T09:50:44.355Z] [2026-07-09T09:50:44.355Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T09:50:44.355Z] ] *** [2026-07-09T09:50:44.355Z] ok: [an-a01n02] [2026-07-09T09:50:44.355Z] ok: [an-a01n01] [2026-07-09T09:50:44.355Z] [2026-07-09T09:50:44.355Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T09:50:44.355Z] ] *** [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (357 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (356 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (355 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (354 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (352 retries left). [2026-07-09T09:52:22.900Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (353 retries left). [2026-07-09T09:52:22.900Z] ok: [an-a01n02] [2026-07-09T09:52:22.900Z] ok: [an-a01n01] [2026-07-09T09:52:22.900Z] [2026-07-09T09:52:22.900Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T09:52:22.900Z] ] *** [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-07-09T09:54:01.440Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-07-09T09:54:01.440Z] ok: [an-a01n02] [2026-07-09T09:54:01.440Z] ok: [an-a01n01] [2026-07-09T09:54:01.440Z] [2026-07-09T09:54:01.440Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T09:54:01.440Z] ] *** [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-07-09T09:55:55.098Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-07-09T09:55:55.098Z] ok: [an-a01n02] [2026-07-09T09:55:55.098Z] ok: [an-a01n01] [2026-07-09T09:55:55.098Z] [2026-07-09T09:55:55.098Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T09:55:55.098Z] ] *** [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (350 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (349 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (348 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (347 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (346 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (345 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (344 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (344 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (343 retries left). [2026-07-09T09:59:19.310Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (343 retries left). [2026-07-09T09:59:19.310Z] ok: [an-a01n01] [2026-07-09T09:59:19.310Z] ok: [an-a01n02] [2026-07-09T09:59:19.310Z] [2026-07-09T09:59:19.310Z] TASK [fence with servers test _raw_params=../../common/fence_with_servers.yml] *** [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T09:59:19.310Z] [2026-07-09T09:59:19.310Z] TASK [fence with servers iteration 1 testiter={{ testiter }}] ****************** [2026-07-09T09:59:19.310Z] ok: [an-a01n01] [2026-07-09T09:59:19.310Z] ok: [an-a01n02] [2026-07-09T09:59:19.310Z] [2026-07-09T09:59:19.310Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T09:59:19.310Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-07-09T09:59:19.310Z] [2026-07-09T09:59:19.310Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T09:59:19.310Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T09:59:19.310Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T09:59:19.310Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T09:59:19.310Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T09:59:19.310Z] echo "pcs:" [2026-07-09T09:59:19.310Z] echo "$pcsstatus" [2026-07-09T09:59:19.310Z] exit 1 [2026-07-09T09:59:19.310Z] fi [2026-07-09T09:59:19.310Z] exit 0 [2026-07-09T09:59:19.310Z] ] *** [2026-07-09T09:59:19.310Z] ok: [an-a01n02] [2026-07-09T09:59:19.310Z] ok: [an-a01n01] [2026-07-09T09:59:19.310Z] [2026-07-09T09:59:19.310Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T09:59:19.310Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T09:59:19.310Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T09:59:19.310Z] ret=0 [2026-07-09T09:59:19.310Z] if [ "$servers" != "1" ]; then [2026-07-09T09:59:19.310Z] echo "Not all servers are running on the same node!" [2026-07-09T09:59:19.310Z] echo "$pcsstatus" [2026-07-09T09:59:19.310Z] ret=1 [2026-07-09T09:59:19.310Z] fi [2026-07-09T09:59:19.310Z] echo "$hypervisor" [2026-07-09T09:59:19.310Z] exit $ret [2026-07-09T09:59:19.310Z] ] *** [2026-07-09T09:59:20.052Z] ok: [an-a01n02] [2026-07-09T09:59:20.052Z] ok: [an-a01n01] [2026-07-09T09:59:20.052Z] [2026-07-09T09:59:20.052Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-07-09T09:59:20.052Z] ok: [an-a01n01] [2026-07-09T09:59:20.052Z] ok: [an-a01n02] [2026-07-09T09:59:20.052Z] [2026-07-09T09:59:20.052Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-07-09T09:59:20.052Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T09:59:20.052Z] [2026-07-09T09:59:20.052Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T09:59:20.052Z] ] ********** [2026-07-09T09:59:20.793Z] skipping: [an-a01n02] [2026-07-09T09:59:20.793Z] changed: [an-a01n01] [2026-07-09T09:59:20.793Z] [2026-07-09T09:59:20.793Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T09:59:20.793Z] skipping: [an-a01n01] [2026-07-09T09:59:20.793Z] skipping: [an-a01n02] [2026-07-09T09:59:20.793Z] [2026-07-09T09:59:20.793Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-07-09T09:59:21.138Z] skipping: [an-a01n01] [2026-07-09T09:59:21.138Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-07-09T09:59:21.138Z] [2026-07-09T09:59:21.138Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-07-09T09:59:21.138Z] ] *** [2026-07-09T10:00:30.764Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-07-09T10:00:30.764Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-07-09T10:00:30.764Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-07-09T10:00:30.764Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-07-09T10:00:30.764Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-07-09T10:00:30.764Z] ok: [an-a01n02] [2026-07-09T10:00:30.764Z] [2026-07-09T10:00:30.764Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:00:30.764Z] skipping: [an-a01n01] [2026-07-09T10:00:30.764Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-07-09T10:00:30.764Z] [2026-07-09T10:00:30.764Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:00:30.764Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:00:30.764Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:00:30.764Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:00:30.764Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:00:30.764Z] echo "pcs:" [2026-07-09T10:00:30.764Z] echo "$pcsstatus" [2026-07-09T10:00:30.764Z] exit 1 [2026-07-09T10:00:30.764Z] fi [2026-07-09T10:00:30.764Z] exit 0 [2026-07-09T10:00:30.764Z] ] *** [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-07-09T10:01:54.208Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:01:54.208Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:01:54.208Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:01:54.208Z] ret=0 [2026-07-09T10:01:54.208Z] if [ "$servers" != "1" ]; then [2026-07-09T10:01:54.208Z] echo "Not all servers are running on the same node!" [2026-07-09T10:01:54.208Z] echo "$pcsstatus" [2026-07-09T10:01:54.208Z] ret=1 [2026-07-09T10:01:54.208Z] fi [2026-07-09T10:01:54.208Z] echo "$hypervisor" [2026-07-09T10:01:54.208Z] exit $ret [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] => (item=1) [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] => (item=2) [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] => (item=3) [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] => (item=4) [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] => (item=5) [2026-07-09T10:01:54.208Z] skipping: [an-a01n01] [2026-07-09T10:01:54.208Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-07-09T10:01:54.208Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-07-09T10:01:54.208Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-07-09T10:01:54.208Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-07-09T10:01:54.208Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.208Z] ok: [an-a01n02] [2026-07-09T10:01:54.208Z] [2026-07-09T10:01:54.208Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:01:54.208Z] ] *** [2026-07-09T10:01:54.951Z] ok: [an-a01n02] [2026-07-09T10:01:54.951Z] [2026-07-09T10:01:54.951Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-07-09T10:01:55.296Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:01:55.296Z] [2026-07-09T10:01:55.296Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T10:05:19.599Z] ok: [an-a01n02 -> localhost] [2026-07-09T10:05:19.599Z] ok: [an-a01n01 -> localhost] [2026-07-09T10:05:19.599Z] [2026-07-09T10:05:19.599Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:05:19.599Z] [2026-07-09T10:05:19.599Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:05:19.599Z] ] *** [2026-07-09T10:05:19.599Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:05:19.599Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:05:19.599Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:05:19.599Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:05:19.599Z] [2026-07-09T10:05:19.599Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:05:19.599Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:05:19.599Z] [2026-07-09T10:05:19.599Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:05:19.599Z] ] *** [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (350 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (349 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (348 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (347 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (346 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (345 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (344 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (343 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (342 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (341 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (340 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (339 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (338 retries left). [2026-07-09T10:09:29.097Z] ok: [an-a01n01] [2026-07-09T10:09:29.097Z] ok: [an-a01n02] [2026-07-09T10:09:29.097Z] [2026-07-09T10:09:29.097Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:09:29.097Z] ] *** [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-07-09T10:09:29.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-07-09T10:09:29.097Z] ok: [an-a01n01] [2026-07-09T10:09:29.097Z] ok: [an-a01n02] [2026-07-09T10:09:29.097Z] [2026-07-09T10:09:29.097Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:09:29.097Z] ] *** [2026-07-09T10:09:29.442Z] ok: [an-a01n01] [2026-07-09T10:09:29.442Z] ok: [an-a01n02] [2026-07-09T10:09:29.442Z] [2026-07-09T10:09:29.442Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:09:29.442Z] ] *** [2026-07-09T10:09:30.184Z] ok: [an-a01n01] [2026-07-09T10:09:30.184Z] ok: [an-a01n02] [2026-07-09T10:09:30.184Z] [2026-07-09T10:09:30.184Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:09:30.184Z] ] *** [2026-07-09T10:09:30.925Z] ok: [an-a01n01] [2026-07-09T10:09:30.925Z] ok: [an-a01n02] [2026-07-09T10:09:30.925Z] [2026-07-09T10:09:30.925Z] TASK [fence with servers iteration 2 testiter={{ testiter }}] ****************** [2026-07-09T10:09:30.925Z] ok: [an-a01n01] [2026-07-09T10:09:30.925Z] ok: [an-a01n02] [2026-07-09T10:09:30.925Z] [2026-07-09T10:09:30.925Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:09:30.925Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:09:30.925Z] [2026-07-09T10:09:30.925Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:09:30.925Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:09:30.925Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:09:30.925Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:09:30.925Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:09:30.925Z] echo "pcs:" [2026-07-09T10:09:30.925Z] echo "$pcsstatus" [2026-07-09T10:09:30.925Z] exit 1 [2026-07-09T10:09:30.925Z] fi [2026-07-09T10:09:30.925Z] exit 0 [2026-07-09T10:09:30.925Z] ] *** [2026-07-09T10:09:33.262Z] ok: [an-a01n01] [2026-07-09T10:09:33.262Z] ok: [an-a01n02] [2026-07-09T10:09:33.262Z] [2026-07-09T10:09:33.262Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:09:33.262Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:09:33.262Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:09:33.262Z] ret=0 [2026-07-09T10:09:33.262Z] if [ "$servers" != "1" ]; then [2026-07-09T10:09:33.262Z] echo "Not all servers are running on the same node!" [2026-07-09T10:09:33.262Z] echo "$pcsstatus" [2026-07-09T10:09:33.262Z] ret=1 [2026-07-09T10:09:33.262Z] fi [2026-07-09T10:09:33.262Z] echo "$hypervisor" [2026-07-09T10:09:33.262Z] exit $ret [2026-07-09T10:09:33.262Z] ] *** [2026-07-09T10:09:34.986Z] ok: [an-a01n01] [2026-07-09T10:09:34.986Z] ok: [an-a01n02] [2026-07-09T10:09:34.986Z] [2026-07-09T10:09:34.986Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-07-09T10:09:34.986Z] ok: [an-a01n01] [2026-07-09T10:09:34.986Z] ok: [an-a01n02] [2026-07-09T10:09:34.986Z] [2026-07-09T10:09:34.986Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-07-09T10:09:34.986Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:09:34.986Z] [2026-07-09T10:09:34.986Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T10:09:34.986Z] ] ********** [2026-07-09T10:09:35.727Z] skipping: [an-a01n01] [2026-07-09T10:09:35.727Z] changed: [an-a01n02] [2026-07-09T10:09:35.727Z] [2026-07-09T10:09:35.727Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T10:09:35.727Z] skipping: [an-a01n01] [2026-07-09T10:09:35.727Z] skipping: [an-a01n02] [2026-07-09T10:09:35.727Z] [2026-07-09T10:09:35.727Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-07-09T10:09:36.073Z] skipping: [an-a01n02] [2026-07-09T10:09:36.073Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-07-09T10:09:36.073Z] [2026-07-09T10:09:36.073Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-07-09T10:09:36.073Z] ] *** [2026-07-09T10:10:45.709Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-07-09T10:10:45.709Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-07-09T10:10:45.709Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-07-09T10:10:45.709Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-07-09T10:10:45.709Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-07-09T10:10:45.709Z] ok: [an-a01n01] [2026-07-09T10:10:45.709Z] [2026-07-09T10:10:45.709Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:10:45.709Z] skipping: [an-a01n02] [2026-07-09T10:10:45.709Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-07-09T10:10:45.709Z] [2026-07-09T10:10:45.709Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:10:45.709Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:10:45.709Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:10:45.709Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:10:45.709Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:10:45.709Z] echo "pcs:" [2026-07-09T10:10:45.709Z] echo "$pcsstatus" [2026-07-09T10:10:45.709Z] exit 1 [2026-07-09T10:10:45.709Z] fi [2026-07-09T10:10:45.709Z] exit 0 [2026-07-09T10:10:45.709Z] ] *** [2026-07-09T10:11:43.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-07-09T10:11:43.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-07-09T10:11:43.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-07-09T10:11:43.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-07-09T10:11:43.766Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-07-09T10:11:43.766Z] ok: [an-a01n01] [2026-07-09T10:11:43.766Z] [2026-07-09T10:11:43.767Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:11:43.767Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:11:43.767Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:11:43.767Z] ret=0 [2026-07-09T10:11:43.767Z] if [ "$servers" != "1" ]; then [2026-07-09T10:11:43.767Z] echo "Not all servers are running on the same node!" [2026-07-09T10:11:43.767Z] echo "$pcsstatus" [2026-07-09T10:11:43.767Z] ret=1 [2026-07-09T10:11:43.767Z] fi [2026-07-09T10:11:43.767Z] echo "$hypervisor" [2026-07-09T10:11:43.767Z] exit $ret [2026-07-09T10:11:43.767Z] ] *** [2026-07-09T10:11:43.767Z] ok: [an-a01n01] [2026-07-09T10:11:43.767Z] [2026-07-09T10:11:43.767Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] => (item=1) [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] => (item=2) [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] => (item=3) [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] => (item=4) [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] => (item=5) [2026-07-09T10:11:43.767Z] skipping: [an-a01n02] [2026-07-09T10:11:43.767Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-07-09T10:11:43.767Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-07-09T10:11:43.767Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-07-09T10:11:43.767Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-07-09T10:11:43.767Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-07-09T10:11:43.767Z] [2026-07-09T10:11:43.767Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:11:43.767Z] ] *** [2026-07-09T10:11:57.112Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-07-09T10:11:57.112Z] ok: [an-a01n01] [2026-07-09T10:11:57.112Z] [2026-07-09T10:11:57.112Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:11:57.112Z] ] *** [2026-07-09T10:11:57.855Z] ok: [an-a01n01] [2026-07-09T10:11:57.855Z] [2026-07-09T10:11:57.855Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:11:57.855Z] ] *** [2026-07-09T10:11:58.669Z] ok: [an-a01n01] [2026-07-09T10:11:58.669Z] [2026-07-09T10:11:58.669Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:11:58.669Z] ] *** [2026-07-09T10:11:59.410Z] ok: [an-a01n01] [2026-07-09T10:11:59.410Z] [2026-07-09T10:11:59.410Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:11:59.410Z] ] *** [2026-07-09T10:12:00.154Z] ok: [an-a01n01] [2026-07-09T10:12:00.154Z] [2026-07-09T10:12:00.154Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-07-09T10:12:00.501Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:12:00.501Z] [2026-07-09T10:12:00.501Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T10:15:25.021Z] ok: [an-a01n01 -> localhost] [2026-07-09T10:15:25.021Z] ok: [an-a01n02 -> localhost] [2026-07-09T10:15:25.021Z] [2026-07-09T10:15:25.021Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:15:25.021Z] [2026-07-09T10:15:25.021Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:15:25.021Z] ] *** [2026-07-09T10:15:25.021Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:15:25.021Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:15:25.021Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:15:25.021Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:15:25.021Z] [2026-07-09T10:15:25.021Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:15:25.021Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:15:25.021Z] [2026-07-09T10:15:25.021Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:15:25.021Z] ] *** [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (360 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (359 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (358 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (357 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (356 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (355 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (354 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (353 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (352 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-07-09T10:17:18.672Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced (351 retries left). [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [fence with servers iteration 3 testiter={{ testiter }}] ****************** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:17:18.672Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:17:18.672Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:17:18.672Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:17:18.672Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:17:18.672Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:17:18.672Z] echo "pcs:" [2026-07-09T10:17:18.672Z] echo "$pcsstatus" [2026-07-09T10:17:18.672Z] exit 1 [2026-07-09T10:17:18.672Z] fi [2026-07-09T10:17:18.672Z] exit 0 [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:17:18.672Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:17:18.672Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:17:18.672Z] ret=0 [2026-07-09T10:17:18.672Z] if [ "$servers" != "1" ]; then [2026-07-09T10:17:18.672Z] echo "Not all servers are running on the same node!" [2026-07-09T10:17:18.672Z] echo "$pcsstatus" [2026-07-09T10:17:18.672Z] ret=1 [2026-07-09T10:17:18.672Z] fi [2026-07-09T10:17:18.672Z] echo "$hypervisor" [2026-07-09T10:17:18.672Z] exit $ret [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-07-09T10:17:18.672Z] ok: [an-a01n01] [2026-07-09T10:17:18.672Z] ok: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-07-09T10:17:18.672Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T10:17:18.672Z] ] ********** [2026-07-09T10:17:18.672Z] skipping: [an-a01n02] [2026-07-09T10:17:18.672Z] changed: [an-a01n01] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T10:17:18.672Z] skipping: [an-a01n01] [2026-07-09T10:17:18.672Z] skipping: [an-a01n02] [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-07-09T10:17:18.672Z] skipping: [an-a01n01] [2026-07-09T10:17:18.672Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-07-09T10:17:18.672Z] [2026-07-09T10:17:18.672Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-07-09T10:17:18.672Z] ] *** [2026-07-09T10:18:16.722Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-07-09T10:18:16.722Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-07-09T10:18:16.722Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-07-09T10:18:16.722Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-07-09T10:18:16.722Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-07-09T10:18:16.722Z] ok: [an-a01n02] [2026-07-09T10:18:16.722Z] [2026-07-09T10:18:16.722Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:18:16.722Z] skipping: [an-a01n01] [2026-07-09T10:18:16.722Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-07-09T10:18:16.722Z] [2026-07-09T10:18:16.722Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:18:16.722Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:18:16.722Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:18:16.722Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:18:16.722Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:18:16.722Z] echo "pcs:" [2026-07-09T10:18:16.722Z] echo "$pcsstatus" [2026-07-09T10:18:16.722Z] exit 1 [2026-07-09T10:18:16.722Z] fi [2026-07-09T10:18:16.722Z] exit 0 [2026-07-09T10:18:16.722Z] ] *** [2026-07-09T10:19:05.137Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-07-09T10:19:05.137Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-07-09T10:19:05.137Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-07-09T10:19:05.137Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-07-09T10:19:05.137Z] ok: [an-a01n02] [2026-07-09T10:19:05.137Z] [2026-07-09T10:19:05.137Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:19:05.137Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:19:05.137Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:19:05.137Z] ret=0 [2026-07-09T10:19:05.137Z] if [ "$servers" != "1" ]; then [2026-07-09T10:19:05.137Z] echo "Not all servers are running on the same node!" [2026-07-09T10:19:05.137Z] echo "$pcsstatus" [2026-07-09T10:19:05.137Z] ret=1 [2026-07-09T10:19:05.137Z] fi [2026-07-09T10:19:05.137Z] echo "$hypervisor" [2026-07-09T10:19:05.137Z] exit $ret [2026-07-09T10:19:05.137Z] ] *** [2026-07-09T10:19:07.475Z] ok: [an-a01n02] [2026-07-09T10:19:07.475Z] [2026-07-09T10:19:07.475Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] => (item=1) [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] => (item=2) [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] => (item=3) [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] => (item=4) [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] => (item=5) [2026-07-09T10:19:07.475Z] skipping: [an-a01n01] [2026-07-09T10:19:07.475Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-07-09T10:19:07.475Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-07-09T10:19:07.475Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-07-09T10:19:07.475Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-07-09T10:19:07.475Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-07-09T10:19:07.475Z] [2026-07-09T10:19:07.475Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:19:07.475Z] ] *** [2026-07-09T10:19:23.574Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-07-09T10:19:23.574Z] ok: [an-a01n02] [2026-07-09T10:19:23.574Z] [2026-07-09T10:19:23.574Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:19:23.574Z] ] *** [2026-07-09T10:19:23.574Z] ok: [an-a01n02] [2026-07-09T10:19:23.574Z] [2026-07-09T10:19:23.574Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:19:23.574Z] ] *** [2026-07-09T10:19:23.920Z] ok: [an-a01n02] [2026-07-09T10:19:23.920Z] [2026-07-09T10:19:23.920Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:19:23.920Z] ] *** [2026-07-09T10:19:24.663Z] ok: [an-a01n02] [2026-07-09T10:19:24.663Z] [2026-07-09T10:19:24.663Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:19:24.663Z] ] *** [2026-07-09T10:19:27.002Z] ok: [an-a01n02] [2026-07-09T10:19:27.002Z] [2026-07-09T10:19:27.002Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-07-09T10:19:27.002Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:19:27.002Z] [2026-07-09T10:19:27.002Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01 -> localhost] [2026-07-09T10:22:51.271Z] ok: [an-a01n02 -> localhost] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:22:51.271Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:22:51.271Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:22:51.271Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [fence with servers iteration 4 testiter={{ testiter }}] ****************** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:22:51.271Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:22:51.271Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:22:51.271Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:22:51.271Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:22:51.271Z] echo "pcs:" [2026-07-09T10:22:51.271Z] echo "$pcsstatus" [2026-07-09T10:22:51.271Z] exit 1 [2026-07-09T10:22:51.271Z] fi [2026-07-09T10:22:51.271Z] exit 0 [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:22:51.271Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:22:51.271Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:22:51.271Z] ret=0 [2026-07-09T10:22:51.271Z] if [ "$servers" != "1" ]; then [2026-07-09T10:22:51.271Z] echo "Not all servers are running on the same node!" [2026-07-09T10:22:51.271Z] echo "$pcsstatus" [2026-07-09T10:22:51.271Z] ret=1 [2026-07-09T10:22:51.271Z] fi [2026-07-09T10:22:51.271Z] echo "$hypervisor" [2026-07-09T10:22:51.271Z] exit $ret [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-07-09T10:22:51.271Z] ok: [an-a01n01] [2026-07-09T10:22:51.271Z] ok: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [kill node "an-a01n02" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T10:22:51.271Z] ] ********** [2026-07-09T10:22:51.271Z] skipping: [an-a01n01] [2026-07-09T10:22:51.271Z] changed: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait for node "an-a01n02" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T10:22:51.271Z] skipping: [an-a01n01] [2026-07-09T10:22:51.271Z] skipping: [an-a01n02] [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-07-09T10:22:51.271Z] skipping: [an-a01n02] [2026-07-09T10:22:51.271Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-07-09T10:22:51.271Z] [2026-07-09T10:22:51.271Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-07-09T10:22:51.271Z] ] *** [2026-07-09T10:23:52.311Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-07-09T10:23:52.311Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-07-09T10:23:52.311Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-07-09T10:23:52.311Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-07-09T10:23:52.311Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-07-09T10:23:52.311Z] ok: [an-a01n01] [2026-07-09T10:23:52.311Z] [2026-07-09T10:23:52.311Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:23:52.311Z] skipping: [an-a01n02] [2026-07-09T10:23:52.311Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-07-09T10:23:52.311Z] [2026-07-09T10:23:52.311Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:23:52.311Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:23:52.311Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:23:52.311Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:23:52.311Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:23:52.311Z] echo "pcs:" [2026-07-09T10:23:52.311Z] echo "$pcsstatus" [2026-07-09T10:23:52.311Z] exit 1 [2026-07-09T10:23:52.311Z] fi [2026-07-09T10:23:52.311Z] exit 0 [2026-07-09T10:23:52.311Z] ] *** [2026-07-09T10:25:01.909Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-07-09T10:25:01.909Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-07-09T10:25:01.909Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-07-09T10:25:01.909Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-07-09T10:25:01.909Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-07-09T10:25:01.909Z] ok: [an-a01n01] [2026-07-09T10:25:01.909Z] [2026-07-09T10:25:01.909Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:25:01.909Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:25:01.909Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:25:01.909Z] ret=0 [2026-07-09T10:25:01.909Z] if [ "$servers" != "1" ]; then [2026-07-09T10:25:01.909Z] echo "Not all servers are running on the same node!" [2026-07-09T10:25:01.909Z] echo "$pcsstatus" [2026-07-09T10:25:01.909Z] ret=1 [2026-07-09T10:25:01.909Z] fi [2026-07-09T10:25:01.909Z] echo "$hypervisor" [2026-07-09T10:25:01.909Z] exit $ret [2026-07-09T10:25:01.909Z] ] *** [2026-07-09T10:25:01.909Z] ok: [an-a01n01] [2026-07-09T10:25:01.909Z] [2026-07-09T10:25:01.909Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] => (item=1) [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] => (item=2) [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] => (item=3) [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] => (item=4) [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] => (item=5) [2026-07-09T10:25:01.909Z] skipping: [an-a01n02] [2026-07-09T10:25:01.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-07-09T10:25:01.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-07-09T10:25:01.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-07-09T10:25:01.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-07-09T10:25:01.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-07-09T10:25:01.909Z] [2026-07-09T10:25:01.909Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:25:01.909Z] ] *** [2026-07-09T10:25:01.909Z] ok: [an-a01n01] [2026-07-09T10:25:01.909Z] [2026-07-09T10:25:01.909Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:25:01.909Z] ] *** [2026-07-09T10:25:15.245Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy2 to respond to ssh (60 retries left). [2026-07-09T10:25:15.245Z] ok: [an-a01n01] [2026-07-09T10:25:15.245Z] [2026-07-09T10:25:15.245Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:25:15.245Z] ] *** [2026-07-09T10:25:15.591Z] ok: [an-a01n01] [2026-07-09T10:25:15.591Z] [2026-07-09T10:25:15.591Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:25:15.591Z] ] *** [2026-07-09T10:25:16.357Z] ok: [an-a01n01] [2026-07-09T10:25:16.357Z] [2026-07-09T10:25:16.357Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:25:16.357Z] ] *** [2026-07-09T10:25:17.098Z] ok: [an-a01n01] [2026-07-09T10:25:17.098Z] [2026-07-09T10:25:17.098Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-07-09T10:25:17.098Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:25:17.098Z] [2026-07-09T10:25:17.098Z] TASK [wait for node "an-a01n02" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n02 -> localhost] [2026-07-09T10:28:41.304Z] ok: [an-a01n01 -> localhost] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:28:41.304Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:28:41.304Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:28:41.304Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [fence with servers iteration 5 testiter={{ testiter }}] ****************** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:28:41.304Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:28:41.304Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:28:41.304Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:28:41.304Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:28:41.304Z] echo "pcs:" [2026-07-09T10:28:41.304Z] echo "$pcsstatus" [2026-07-09T10:28:41.304Z] exit 1 [2026-07-09T10:28:41.304Z] fi [2026-07-09T10:28:41.304Z] exit 0 [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:28:41.304Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:28:41.304Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:28:41.304Z] ret=0 [2026-07-09T10:28:41.304Z] if [ "$servers" != "1" ]; then [2026-07-09T10:28:41.304Z] echo "Not all servers are running on the same node!" [2026-07-09T10:28:41.304Z] echo "$pcsstatus" [2026-07-09T10:28:41.304Z] ret=1 [2026-07-09T10:28:41.304Z] fi [2026-07-09T10:28:41.304Z] echo "$hypervisor" [2026-07-09T10:28:41.304Z] exit $ret [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [set target target={{ hypervisor.stdout_lines[-1] }}] ********************* [2026-07-09T10:28:41.304Z] ok: [an-a01n01] [2026-07-09T10:28:41.304Z] ok: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [will kill node with servers _raw_params=./kill_node.yml] ***************** [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [kill node "an-a01n01" _raw_params=echo c > /proc/sysrq-trigger [2026-07-09T10:28:41.304Z] ] ********** [2026-07-09T10:28:41.304Z] skipping: [an-a01n02] [2026-07-09T10:28:41.304Z] changed: [an-a01n01] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait for node "an-a01n01" to be back online _raw_params=./wait_node.yml] *** [2026-07-09T10:28:41.304Z] skipping: [an-a01n01] [2026-07-09T10:28:41.304Z] skipping: [an-a01n02] [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait for node to be offline _raw_params=./offline_node.yml] ************** [2026-07-09T10:28:41.304Z] skipping: [an-a01n01] [2026-07-09T10:28:41.304Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-07-09T10:28:41.304Z] [2026-07-09T10:28:41.304Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline _raw_params=pcs status | grep OFFLINE | grep -q "{{ offlinenode }}" [2026-07-09T10:28:41.304Z] ] *** [2026-07-09T10:29:50.909Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-07-09T10:29:50.909Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-07-09T10:29:50.909Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-07-09T10:29:50.909Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-07-09T10:29:50.909Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-07-09T10:29:50.909Z] ok: [an-a01n02] [2026-07-09T10:29:50.909Z] [2026-07-09T10:29:50.909Z] TASK [check servers status _raw_params=./server_started.yml] ******************* [2026-07-09T10:29:50.909Z] skipping: [an-a01n01] [2026-07-09T10:29:50.909Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-07-09T10:29:50.909Z] [2026-07-09T10:29:50.909Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:29:50.909Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:29:50.909Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:29:50.909Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:29:50.909Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:29:50.909Z] echo "pcs:" [2026-07-09T10:29:50.909Z] echo "$pcsstatus" [2026-07-09T10:29:50.909Z] exit 1 [2026-07-09T10:29:50.909Z] fi [2026-07-09T10:29:50.909Z] exit 0 [2026-07-09T10:29:50.909Z] ] *** [2026-07-09T10:30:48.991Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-07-09T10:30:48.991Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-07-09T10:30:48.991Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-07-09T10:30:48.991Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-07-09T10:30:48.991Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-07-09T10:30:48.991Z] ok: [an-a01n02] [2026-07-09T10:30:48.991Z] [2026-07-09T10:30:48.991Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:30:48.991Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:30:48.991Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:30:48.991Z] ret=0 [2026-07-09T10:30:48.991Z] if [ "$servers" != "1" ]; then [2026-07-09T10:30:48.991Z] echo "Not all servers are running on the same node!" [2026-07-09T10:30:48.991Z] echo "$pcsstatus" [2026-07-09T10:30:48.991Z] ret=1 [2026-07-09T10:30:48.991Z] fi [2026-07-09T10:30:48.991Z] echo "$hypervisor" [2026-07-09T10:30:48.991Z] exit $ret [2026-07-09T10:30:48.991Z] ] *** [2026-07-09T10:30:48.991Z] ok: [an-a01n02] [2026-07-09T10:30:48.991Z] [2026-07-09T10:30:48.991Z] TASK [wait servers to respond to ssh _raw_params=./wait_server_ssh.yml] ******** [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] => (item=1) [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] => (item=2) [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] => (item=3) [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] => (item=4) [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] => (item=5) [2026-07-09T10:30:48.991Z] skipping: [an-a01n01] [2026-07-09T10:30:48.991Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-07-09T10:30:48.991Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-07-09T10:30:48.991Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-07-09T10:30:48.991Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-07-09T10:30:48.991Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-07-09T10:30:48.991Z] [2026-07-09T10:30:48.991Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:30:48.991Z] ] *** [2026-07-09T10:31:05.091Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-07-09T10:31:05.091Z] ok: [an-a01n02] [2026-07-09T10:31:05.091Z] [2026-07-09T10:31:05.091Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:31:05.091Z] ] *** [2026-07-09T10:31:05.091Z] ok: [an-a01n02] [2026-07-09T10:31:05.091Z] [2026-07-09T10:31:05.091Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:31:05.091Z] ] *** [2026-07-09T10:31:05.091Z] ok: [an-a01n02] [2026-07-09T10:31:05.091Z] [2026-07-09T10:31:05.091Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:31:05.091Z] ] *** [2026-07-09T10:31:05.091Z] ok: [an-a01n02] [2026-07-09T10:31:05.091Z] [2026-07-09T10:31:05.091Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T10:31:05.091Z] ] *** [2026-07-09T10:31:05.833Z] ok: [an-a01n02] [2026-07-09T10:31:05.833Z] [2026-07-09T10:31:05.833Z] TASK [wait for node to be back online _raw_params=./wait_node.yml] ************* [2026-07-09T10:31:05.833Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-07-09T10:31:05.833Z] [2026-07-09T10:31:05.833Z] TASK [wait for node "an-a01n01" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n01 -> localhost] [2026-07-09T10:34:30.111Z] ok: [an-a01n02 -> localhost] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [check cluster is in good state _raw_params=./cluster_good.yml] *********** [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:34:30.111Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:34:30.111Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:34:30.111Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [wait servers drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] PLAY [basic server tests] ****************************************************** [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:34:30.111Z] ok: [an-a01n01] [2026-07-09T10:34:30.111Z] ok: [an-a01n02] [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T10:34:30.111Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:30.111Z] [2026-07-09T10:34:30.111Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:34:30.111Z] ] *** [2026-07-09T10:34:30.852Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:34:30.852Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:34:30.852Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:34:30.852Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:34:30.852Z] [2026-07-09T10:34:30.852Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-07-09T10:34:30.852Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:30.852Z] [2026-07-09T10:34:30.852Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-07-09T10:34:30.852Z] if [ -z "$storage" ]; then [2026-07-09T10:34:30.852Z] echo "Unable to detect shared storage" [2026-07-09T10:34:30.852Z] exit 1 [2026-07-09T10:34:30.852Z] fi [2026-07-09T10:34:30.852Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:34:30.852Z] ] *** [2026-07-09T10:34:36.907Z] changed: [an-a01n01] [2026-07-09T10:34:36.907Z] changed: [an-a01n02] [2026-07-09T10:34:36.907Z] [2026-07-09T10:34:36.907Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-07-09T10:34:36.907Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:36.907Z] [2026-07-09T10:34:36.907Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-07-09T10:34:36.907Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-07-09T10:34:36.907Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-07-09T10:34:36.907Z] echo "Unable to detect all deploy iso" [2026-07-09T10:34:36.907Z] echo "Detected isos" [2026-07-09T10:34:36.907Z] echo "$isos" [2026-07-09T10:34:36.907Z] exit 1 [2026-07-09T10:34:36.907Z] fi [2026-07-09T10:34:36.907Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-07-09T10:34:36.907Z] if [ -z "$installiso" ]; then [2026-07-09T10:34:36.907Z] echo "Unable to detect install iso" [2026-07-09T10:34:36.907Z] echo "Detected isos" [2026-07-09T10:34:36.907Z] echo "$isos" [2026-07-09T10:34:36.907Z] exit 1 [2026-07-09T10:34:36.907Z] fi [2026-07-09T10:34:36.907Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:34:36.907Z] ] *** [2026-07-09T10:34:42.958Z] changed: [an-a01n01] [2026-07-09T10:34:42.958Z] changed: [an-a01n02] [2026-07-09T10:34:42.958Z] [2026-07-09T10:34:42.958Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-07-09T10:34:42.958Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:42.958Z] [2026-07-09T10:34:42.958Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:34:42.958Z] ] *** [2026-07-09T10:34:43.707Z] skipping: [an-a01n02] [2026-07-09T10:34:43.707Z] ok: [an-a01n01] [2026-07-09T10:34:43.707Z] [2026-07-09T10:34:43.707Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T10:34:43.707Z] --ci-test \ [2026-07-09T10:34:43.707Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T10:34:43.707Z] --os almalinux9 \ [2026-07-09T10:34:43.707Z] --cpu 2 --ram 4G \ [2026-07-09T10:34:43.707Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T10:34:43.707Z] --install-media {{ installisouuid }} \ [2026-07-09T10:34:43.707Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T10:34:43.707Z] ] *** [2026-07-09T10:34:43.707Z] skipping: [an-a01n01] [2026-07-09T10:34:43.707Z] skipping: [an-a01n02] [2026-07-09T10:34:43.707Z] [2026-07-09T10:34:43.707Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T10:34:43.707Z] skipping: [an-a01n01] [2026-07-09T10:34:43.707Z] skipping: [an-a01n02] [2026-07-09T10:34:43.707Z] [2026-07-09T10:34:43.707Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:34:44.057Z] skipping: [an-a01n01] [2026-07-09T10:34:44.057Z] skipping: [an-a01n02] [2026-07-09T10:34:44.057Z] [2026-07-09T10:34:44.057Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T10:34:44.057Z] skipping: [an-a01n01] [2026-07-09T10:34:44.057Z] skipping: [an-a01n02] [2026-07-09T10:34:44.057Z] [2026-07-09T10:34:44.057Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T10:34:44.057Z] skipping: [an-a01n01] [2026-07-09T10:34:44.057Z] skipping: [an-a01n02] [2026-07-09T10:34:44.057Z] [2026-07-09T10:34:44.057Z] TASK [deploy server 2 from "an-a01n02" _raw_params=../../common/deploy_server.yml] *** [2026-07-09T10:34:44.057Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:44.057Z] [2026-07-09T10:34:44.057Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:34:44.057Z] ] *** [2026-07-09T10:34:45.782Z] skipping: [an-a01n01] [2026-07-09T10:34:45.782Z] ok: [an-a01n02] [2026-07-09T10:34:45.782Z] [2026-07-09T10:34:45.782Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T10:34:45.782Z] --ci-test \ [2026-07-09T10:34:45.782Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T10:34:45.782Z] --os almalinux9 \ [2026-07-09T10:34:45.782Z] --cpu 2 --ram 4G \ [2026-07-09T10:34:45.782Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T10:34:45.782Z] --install-media {{ installisouuid }} \ [2026-07-09T10:34:45.782Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T10:34:45.782Z] ] *** [2026-07-09T10:34:45.782Z] skipping: [an-a01n01] [2026-07-09T10:34:45.782Z] skipping: [an-a01n02] [2026-07-09T10:34:45.782Z] [2026-07-09T10:34:45.782Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T10:34:45.782Z] skipping: [an-a01n01] [2026-07-09T10:34:45.782Z] skipping: [an-a01n02] [2026-07-09T10:34:45.782Z] [2026-07-09T10:34:45.782Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:34:46.128Z] skipping: [an-a01n01] [2026-07-09T10:34:46.128Z] skipping: [an-a01n02] [2026-07-09T10:34:46.128Z] [2026-07-09T10:34:46.128Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T10:34:46.128Z] skipping: [an-a01n01] [2026-07-09T10:34:46.128Z] skipping: [an-a01n02] [2026-07-09T10:34:46.128Z] [2026-07-09T10:34:46.128Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T10:34:46.128Z] skipping: [an-a01n01] [2026-07-09T10:34:46.128Z] skipping: [an-a01n02] [2026-07-09T10:34:46.128Z] [2026-07-09T10:34:46.128Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-07-09T10:34:46.128Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:46.128Z] [2026-07-09T10:34:46.128Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:34:46.128Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:34:46.128Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:34:46.128Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:34:46.128Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:34:46.128Z] echo "pcs:" [2026-07-09T10:34:46.128Z] echo "$pcsstatus" [2026-07-09T10:34:46.128Z] exit 1 [2026-07-09T10:34:46.128Z] fi [2026-07-09T10:34:46.128Z] exit 0 [2026-07-09T10:34:46.128Z] ] *** [2026-07-09T10:34:48.468Z] ok: [an-a01n01] [2026-07-09T10:34:48.468Z] ok: [an-a01n02] [2026-07-09T10:34:48.468Z] [2026-07-09T10:34:48.468Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:34:48.468Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:34:48.468Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:34:48.468Z] ret=0 [2026-07-09T10:34:48.468Z] if [ "$servers" != "1" ]; then [2026-07-09T10:34:48.468Z] echo "Not all servers are running on the same node!" [2026-07-09T10:34:48.468Z] echo "$pcsstatus" [2026-07-09T10:34:48.468Z] ret=1 [2026-07-09T10:34:48.468Z] fi [2026-07-09T10:34:48.468Z] echo "$hypervisor" [2026-07-09T10:34:48.468Z] exit $ret [2026-07-09T10:34:48.468Z] ] *** [2026-07-09T10:34:50.822Z] ok: [an-a01n01] [2026-07-09T10:34:50.822Z] ok: [an-a01n02] [2026-07-09T10:34:50.822Z] [2026-07-09T10:34:50.822Z] PLAY [delete extra server tests] *********************************************** [2026-07-09T10:34:50.822Z] [2026-07-09T10:34:50.822Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:34:52.553Z] ok: [an-a01n01] [2026-07-09T10:34:52.553Z] ok: [an-a01n02] [2026-07-09T10:34:52.553Z] [2026-07-09T10:34:52.553Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T10:34:52.899Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:34:52.899Z] [2026-07-09T10:34:52.899Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:34:52.899Z] ] *** [2026-07-09T10:34:56.794Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:34:56.794Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:34:56.794Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:34:56.794Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:34:56.794Z] [2026-07-09T10:34:56.794Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-07-09T10:34:56.794Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T10:34:56.794Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T10:34:56.794Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T10:34:56.794Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T10:34:56.794Z] [2026-07-09T10:34:56.794Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:34:56.794Z] ] *** [2026-07-09T10:34:58.519Z] skipping: [an-a01n01] [2026-07-09T10:34:58.519Z] ok: [an-a01n02] [2026-07-09T10:34:58.519Z] [2026-07-09T10:34:58.519Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T10:34:58.519Z] --force \ [2026-07-09T10:34:58.519Z] --server an-test-deploy{{ servernum }} [2026-07-09T10:34:58.519Z] ] *** [2026-07-09T10:35:04.569Z] skipping: [an-a01n01] [2026-07-09T10:35:04.569Z] changed: [an-a01n02] [2026-07-09T10:35:04.569Z] [2026-07-09T10:35:04.569Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:35:04.569Z] ] *** [2026-07-09T10:35:32.611Z] skipping: [an-a01n01] [2026-07-09T10:35:32.611Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-07-09T10:35:32.611Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-07-09T10:35:32.611Z] ok: [an-a01n02] [2026-07-09T10:35:32.611Z] [2026-07-09T10:35:32.611Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T10:35:32.611Z] ] *** [2026-07-09T10:35:32.611Z] skipping: [an-a01n01] [2026-07-09T10:35:32.611Z] ok: [an-a01n02] [2026-07-09T10:35:32.611Z] [2026-07-09T10:35:32.611Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T10:35:32.611Z] ] *** [2026-07-09T10:35:32.611Z] skipping: [an-a01n01] [2026-07-09T10:35:32.611Z] ok: [an-a01n02] [2026-07-09T10:35:32.611Z] [2026-07-09T10:35:32.611Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:35:32.611Z] ] *** [2026-07-09T10:35:32.611Z] skipping: [an-a01n01] [2026-07-09T10:35:32.611Z] ok: [an-a01n02] [2026-07-09T10:35:32.611Z] [2026-07-09T10:35:32.611Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T10:35:32.611Z] --force \ [2026-07-09T10:35:32.611Z] --server an-test-deploy{{ servernum }} [2026-07-09T10:35:32.611Z] ] *** [2026-07-09T10:35:40.043Z] skipping: [an-a01n01] [2026-07-09T10:35:40.043Z] changed: [an-a01n02] [2026-07-09T10:35:40.043Z] [2026-07-09T10:35:40.043Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:35:40.043Z] ] *** [2026-07-09T10:36:49.637Z] skipping: [an-a01n01] [2026-07-09T10:36:49.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-07-09T10:36:49.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-07-09T10:36:49.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-07-09T10:36:49.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-07-09T10:36:49.637Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-07-09T10:36:49.637Z] ok: [an-a01n02] [2026-07-09T10:36:49.637Z] [2026-07-09T10:36:49.637Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T10:36:49.637Z] ] *** [2026-07-09T10:36:49.637Z] skipping: [an-a01n01] [2026-07-09T10:36:49.637Z] ok: [an-a01n02] [2026-07-09T10:36:49.637Z] [2026-07-09T10:36:49.637Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T10:36:49.637Z] ] *** [2026-07-09T10:36:49.637Z] skipping: [an-a01n01] [2026-07-09T10:36:49.637Z] ok: [an-a01n02] [2026-07-09T10:36:49.637Z] [2026-07-09T10:36:49.637Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:36:49.637Z] ] *** [2026-07-09T10:36:49.637Z] skipping: [an-a01n01] [2026-07-09T10:36:49.637Z] ok: [an-a01n02] [2026-07-09T10:36:49.637Z] [2026-07-09T10:36:49.637Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T10:36:49.637Z] --force \ [2026-07-09T10:36:49.637Z] --server an-test-deploy{{ servernum }} [2026-07-09T10:36:49.637Z] ] *** [2026-07-09T10:36:49.637Z] skipping: [an-a01n01] [2026-07-09T10:36:49.637Z] changed: [an-a01n02] [2026-07-09T10:36:49.637Z] [2026-07-09T10:36:49.637Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:36:49.637Z] ] *** [2026-07-09T10:38:13.069Z] skipping: [an-a01n01] [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (55 retries left). [2026-07-09T10:38:13.069Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (54 retries left). [2026-07-09T10:38:13.069Z] ok: [an-a01n02] [2026-07-09T10:38:13.069Z] [2026-07-09T10:38:13.069Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T10:38:13.069Z] ] *** [2026-07-09T10:38:13.069Z] skipping: [an-a01n01] [2026-07-09T10:38:13.069Z] ok: [an-a01n02] [2026-07-09T10:38:13.069Z] [2026-07-09T10:38:13.069Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T10:38:13.069Z] ] *** [2026-07-09T10:38:13.069Z] skipping: [an-a01n01] [2026-07-09T10:38:13.069Z] ok: [an-a01n02] [2026-07-09T10:38:13.069Z] [2026-07-09T10:38:13.069Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:38:13.069Z] ] *** [2026-07-09T10:38:13.414Z] skipping: [an-a01n01] [2026-07-09T10:38:13.414Z] ok: [an-a01n02] [2026-07-09T10:38:13.414Z] [2026-07-09T10:38:13.414Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T10:38:13.414Z] --force \ [2026-07-09T10:38:13.414Z] --server an-test-deploy{{ servernum }} [2026-07-09T10:38:13.414Z] ] *** [2026-07-09T10:38:19.466Z] skipping: [an-a01n01] [2026-07-09T10:38:19.466Z] changed: [an-a01n02] [2026-07-09T10:38:19.466Z] [2026-07-09T10:38:19.466Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:38:19.466Z] ] *** [2026-07-09T10:39:29.061Z] skipping: [an-a01n01] [2026-07-09T10:39:29.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-07-09T10:39:29.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-07-09T10:39:29.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-07-09T10:39:29.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-07-09T10:39:29.061Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-07-09T10:39:29.061Z] ok: [an-a01n02] [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T10:39:29.061Z] ] *** [2026-07-09T10:39:29.061Z] skipping: [an-a01n01] [2026-07-09T10:39:29.061Z] ok: [an-a01n02] [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T10:39:29.061Z] ] *** [2026-07-09T10:39:29.061Z] skipping: [an-a01n01] [2026-07-09T10:39:29.061Z] ok: [an-a01n02] [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] PLAY [create server for DR testing] ******************************************** [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:39:29.061Z] ok: [an-a01n02] [2026-07-09T10:39:29.061Z] ok: [an-a01n01] [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T10:39:29.061Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:39:29.061Z] ] *** [2026-07-09T10:39:29.061Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:39:29.061Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:39:29.061Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:39:29.061Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-07-09T10:39:29.061Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:29.061Z] [2026-07-09T10:39:29.061Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-07-09T10:39:29.061Z] if [ -z "$storage" ]; then [2026-07-09T10:39:29.061Z] echo "Unable to detect shared storage" [2026-07-09T10:39:29.061Z] exit 1 [2026-07-09T10:39:29.061Z] fi [2026-07-09T10:39:29.061Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:39:29.061Z] ] *** [2026-07-09T10:39:31.397Z] changed: [an-a01n02] [2026-07-09T10:39:31.397Z] changed: [an-a01n01] [2026-07-09T10:39:31.397Z] [2026-07-09T10:39:31.397Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-07-09T10:39:31.397Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:31.397Z] [2026-07-09T10:39:31.397Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-07-09T10:39:31.397Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-07-09T10:39:31.397Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-07-09T10:39:31.397Z] echo "Unable to detect all deploy iso" [2026-07-09T10:39:31.397Z] echo "Detected isos" [2026-07-09T10:39:31.397Z] echo "$isos" [2026-07-09T10:39:31.397Z] exit 1 [2026-07-09T10:39:31.397Z] fi [2026-07-09T10:39:31.397Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-07-09T10:39:31.397Z] if [ -z "$installiso" ]; then [2026-07-09T10:39:31.397Z] echo "Unable to detect install iso" [2026-07-09T10:39:31.397Z] echo "Detected isos" [2026-07-09T10:39:31.397Z] echo "$isos" [2026-07-09T10:39:31.397Z] exit 1 [2026-07-09T10:39:31.397Z] fi [2026-07-09T10:39:31.397Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:39:31.397Z] ] *** [2026-07-09T10:39:36.283Z] changed: [an-a01n01] [2026-07-09T10:39:36.283Z] changed: [an-a01n02] [2026-07-09T10:39:36.283Z] [2026-07-09T10:39:36.283Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-07-09T10:39:36.628Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:36.628Z] [2026-07-09T10:39:36.628Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:39:36.628Z] ] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] ok: [an-a01n01] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T10:39:38.350Z] --ci-test \ [2026-07-09T10:39:38.350Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T10:39:38.350Z] --os almalinux9 \ [2026-07-09T10:39:38.350Z] --cpu 2 --ram 4G \ [2026-07-09T10:39:38.350Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T10:39:38.350Z] --install-media {{ installisouuid }} \ [2026-07-09T10:39:38.350Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T10:39:38.350Z] ] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n01] [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n01] [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n01] [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n01] [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T10:39:38.350Z] skipping: [an-a01n01] [2026-07-09T10:39:38.350Z] skipping: [an-a01n02] [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-07-09T10:39:38.350Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:38.350Z] [2026-07-09T10:39:38.350Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:39:38.350Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:39:38.350Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:39:38.350Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:39:38.350Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:39:38.350Z] echo "pcs:" [2026-07-09T10:39:38.350Z] echo "$pcsstatus" [2026-07-09T10:39:38.350Z] exit 1 [2026-07-09T10:39:38.350Z] fi [2026-07-09T10:39:38.350Z] exit 0 [2026-07-09T10:39:38.350Z] ] *** [2026-07-09T10:39:40.685Z] ok: [an-a01n02] [2026-07-09T10:39:40.685Z] ok: [an-a01n01] [2026-07-09T10:39:40.685Z] [2026-07-09T10:39:40.685Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:39:40.685Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:39:40.685Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:39:40.685Z] ret=0 [2026-07-09T10:39:40.685Z] if [ "$servers" != "1" ]; then [2026-07-09T10:39:40.685Z] echo "Not all servers are running on the same node!" [2026-07-09T10:39:40.685Z] echo "$pcsstatus" [2026-07-09T10:39:40.685Z] ret=1 [2026-07-09T10:39:40.685Z] fi [2026-07-09T10:39:40.685Z] echo "$hypervisor" [2026-07-09T10:39:40.685Z] exit $ret [2026-07-09T10:39:40.685Z] ] *** [2026-07-09T10:39:43.021Z] ok: [an-a01n02] [2026-07-09T10:39:43.021Z] ok: [an-a01n01] [2026-07-09T10:39:43.021Z] [2026-07-09T10:39:43.021Z] PLAY [DR tests] **************************************************************** [2026-07-09T10:39:43.021Z] skipping: no hosts matched [2026-07-09T10:39:43.021Z] [2026-07-09T10:39:43.021Z] PLAY [upgrade server test] ***************************************************** [2026-07-09T10:39:43.021Z] [2026-07-09T10:39:43.021Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:39:44.743Z] ok: [an-a01n01] [2026-07-09T10:39:44.743Z] ok: [an-a01n02] [2026-07-09T10:39:44.743Z] [2026-07-09T10:39:44.743Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T10:39:44.743Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:44.743Z] [2026-07-09T10:39:44.743Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T10:39:44.743Z] ] *** [2026-07-09T10:39:48.635Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T10:39:48.635Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T10:39:48.635Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T10:39:48.635Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T10:39:48.635Z] [2026-07-09T10:39:48.635Z] TASK [check anvil-provision-server storage group _raw_params=../../common/get_storage_group.yml] *** [2026-07-09T10:39:48.635Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:48.635Z] [2026-07-09T10:39:48.635Z] TASK [get anvil-provision-server storage group _raw_params=storage="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep storage_group)" [2026-07-09T10:39:48.635Z] if [ -z "$storage" ]; then [2026-07-09T10:39:48.635Z] echo "Unable to detect shared storage" [2026-07-09T10:39:48.635Z] exit 1 [2026-07-09T10:39:48.635Z] fi [2026-07-09T10:39:48.635Z] echo "$storage" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:39:48.635Z] ] *** [2026-07-09T10:39:54.685Z] changed: [an-a01n01] [2026-07-09T10:39:54.685Z] changed: [an-a01n02] [2026-07-09T10:39:54.685Z] [2026-07-09T10:39:54.685Z] TASK [check anvil-provision-server install isos _raw_params=../../common/get_install_iso.yml] *** [2026-07-09T10:39:54.685Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:54.685Z] [2026-07-09T10:39:54.685Z] TASK [check anvil-provision-server install isos _raw_params=isos="$(anvil-provision-server -vv --log-secure --ci-test --options --machine | grep iso_file)" [2026-07-09T10:39:54.685Z] deployiso="$(echo "$isos" | grep "=deploy")" [2026-07-09T10:39:54.685Z] if [ "$(echo "$deployiso" | wc -l)" -lt "5" ]; then [2026-07-09T10:39:54.685Z] echo "Unable to detect all deploy iso" [2026-07-09T10:39:54.685Z] echo "Detected isos" [2026-07-09T10:39:54.685Z] echo "$isos" [2026-07-09T10:39:54.685Z] exit 1 [2026-07-09T10:39:54.685Z] fi [2026-07-09T10:39:54.685Z] installiso="$(echo "$isos" | grep -v "=deploy")" [2026-07-09T10:39:54.685Z] if [ -z "$installiso" ]; then [2026-07-09T10:39:54.685Z] echo "Unable to detect install iso" [2026-07-09T10:39:54.685Z] echo "Detected isos" [2026-07-09T10:39:54.685Z] echo "$isos" [2026-07-09T10:39:54.685Z] exit 1 [2026-07-09T10:39:54.685Z] fi [2026-07-09T10:39:54.685Z] echo "$installiso" | sed -e 's#.*uuid=##g' -e 's#,.*##g' [2026-07-09T10:39:54.685Z] ] *** [2026-07-09T10:39:59.568Z] changed: [an-a01n01] [2026-07-09T10:39:59.568Z] changed: [an-a01n02] [2026-07-09T10:39:59.568Z] [2026-07-09T10:39:59.568Z] TASK [deploy server 1 from "an-a01n01" _raw_params=../../common/deploy_server.yml] *** [2026-07-09T10:39:59.568Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-07-09T10:39:59.568Z] [2026-07-09T10:39:59.568Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T10:39:59.568Z] ] *** [2026-07-09T10:40:01.290Z] skipping: [an-a01n02] [2026-07-09T10:40:01.290Z] ok: [an-a01n01] [2026-07-09T10:40:01.290Z] [2026-07-09T10:40:01.290Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 197dd09e-7c42-44bf-abef-5cd33d1b0788 _raw_params=anvil-provision-server -vv --log-secure \ [2026-07-09T10:40:01.290Z] --ci-test \ [2026-07-09T10:40:01.290Z] --name an-test-deploy{{ servernum }} \ [2026-07-09T10:40:01.290Z] --os almalinux9 \ [2026-07-09T10:40:01.290Z] --cpu 2 --ram 4G \ [2026-07-09T10:40:01.290Z] --storage-group {{ storageuuid }} --storage-size 10G \ [2026-07-09T10:40:01.290Z] --install-media {{ installisouuid }} \ [2026-07-09T10:40:01.290Z] --driver-disc deploy{{ servernum }}.iso [2026-07-09T10:40:01.290Z] ] *** [2026-07-09T10:40:01.290Z] skipping: [an-a01n01] [2026-07-09T10:40:01.290Z] skipping: [an-a01n02] [2026-07-09T10:40:01.290Z] [2026-07-09T10:40:01.290Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=./wait_server_in_pacemaker.yml] *** [2026-07-09T10:40:01.290Z] skipping: [an-a01n01] [2026-07-09T10:40:01.290Z] skipping: [an-a01n02] [2026-07-09T10:40:01.290Z] [2026-07-09T10:40:01.290Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=./wait_server_drbd_sync.yml] *** [2026-07-09T10:40:01.634Z] skipping: [an-a01n01] [2026-07-09T10:40:01.634Z] skipping: [an-a01n02] [2026-07-09T10:40:01.634Z] [2026-07-09T10:40:01.634Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation _raw_params=./wait_server_ping.yml] *** [2026-07-09T10:40:01.634Z] skipping: [an-a01n01] [2026-07-09T10:40:01.634Z] skipping: [an-a01n02] [2026-07-09T10:40:01.634Z] [2026-07-09T10:40:01.634Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation _raw_params=./wait_server_ssh.yml] *** [2026-07-09T10:40:01.634Z] skipping: [an-a01n01] [2026-07-09T10:40:01.634Z] skipping: [an-a01n02] [2026-07-09T10:40:01.634Z] [2026-07-09T10:40:01.634Z] TASK [check servers status _raw_params=../../common/server_started.yml] ******** [2026-07-09T10:40:01.634Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-07-09T10:40:01.634Z] [2026-07-09T10:40:01.634Z] TASK [wait up to 20 minutes for servers to be Started _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:40:01.634Z] numservers="$(echo "$pcsstatus" | wc -l)" [2026-07-09T10:40:01.634Z] startedservers="$(echo "$pcsstatus" | grep Started | wc -l)" [2026-07-09T10:40:01.634Z] if [ "$numservers" != "$startedservers" ]; then [2026-07-09T10:40:01.634Z] echo "Detected $numservers but only $startedservers in Started state" [2026-07-09T10:40:01.634Z] echo "pcs:" [2026-07-09T10:40:01.634Z] echo "$pcsstatus" [2026-07-09T10:40:01.634Z] exit 1 [2026-07-09T10:40:01.634Z] fi [2026-07-09T10:40:01.634Z] exit 0 [2026-07-09T10:40:01.634Z] ] *** [2026-07-09T10:40:03.972Z] ok: [an-a01n02] [2026-07-09T10:40:03.972Z] ok: [an-a01n01] [2026-07-09T10:40:03.972Z] [2026-07-09T10:40:03.972Z] TASK [check servers are started all on the same node _raw_params=pcsstatus="$(pcs status | grep ocf | grep :alteeve:server)" [2026-07-09T10:40:03.972Z] hypervisor="$(echo "$pcsstatus" | sed -e 's#.*Started##g' | awk '{print $1}' | sort -u)" [2026-07-09T10:40:03.972Z] servers="$(echo "$hypervisor" | wc -l)" [2026-07-09T10:40:03.972Z] ret=0 [2026-07-09T10:40:03.972Z] if [ "$servers" != "1" ]; then [2026-07-09T10:40:03.972Z] echo "Not all servers are running on the same node!" [2026-07-09T10:40:03.972Z] echo "$pcsstatus" [2026-07-09T10:40:03.972Z] ret=1 [2026-07-09T10:40:03.972Z] fi [2026-07-09T10:40:03.972Z] echo "$hypervisor" [2026-07-09T10:40:03.972Z] exit $ret [2026-07-09T10:40:03.972Z] ] *** [2026-07-09T10:40:05.695Z] ok: [an-a01n01] [2026-07-09T10:40:05.695Z] ok: [an-a01n02] [2026-07-09T10:40:05.695Z] [2026-07-09T10:40:05.695Z] PLAY [enable upgrade repo] ***************************************************** [2026-07-09T10:40:05.695Z] [2026-07-09T10:40:05.695Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:40:08.038Z] ok: [an-a01n01] [2026-07-09T10:40:08.038Z] ok: [an-a01n02] [2026-07-09T10:40:08.038Z] ok: [an-striker01] [2026-07-09T10:40:08.038Z] ok: [an-striker02] [2026-07-09T10:40:08.038Z] [2026-07-09T10:40:08.038Z] TASK [enable anvil test-update repo _raw_params=updaterepo=$(dnf repolist --all | grep "anvil[_|-]test[_|-]update" | awk '{print $1}') [2026-07-09T10:40:08.038Z] dnf config-manager --set-enabled $updaterepo [2026-07-09T10:40:08.038Z] ] *** [2026-07-09T10:40:10.380Z] changed: [an-a01n02] [2026-07-09T10:40:10.380Z] changed: [an-a01n01] [2026-07-09T10:40:10.380Z] changed: [an-striker02] [2026-07-09T10:40:10.380Z] changed: [an-striker01] [2026-07-09T10:40:10.380Z] [2026-07-09T10:40:10.380Z] TASK [store current uptime uptime_pre_upgrade={{ ansible_facts.uptime_seconds }}] *** [2026-07-09T10:40:10.380Z] ok: [an-striker01] [2026-07-09T10:40:10.380Z] ok: [an-striker02] [2026-07-09T10:40:10.380Z] ok: [an-a01n01] [2026-07-09T10:40:10.380Z] ok: [an-a01n02] [2026-07-09T10:40:10.380Z] [2026-07-09T10:40:10.380Z] PLAY [start upgrade process] *************************************************** [2026-07-09T10:40:10.380Z] [2026-07-09T10:40:10.380Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:40:12.717Z] ok: [an-striker01] [2026-07-09T10:40:12.717Z] [2026-07-09T10:40:12.717Z] TASK [upgrade and reboot all nodes but an-striker01 _raw_params=striker-update-cluster -vv --log-secure --clear-cache --timeout 60m --reboot --yes [2026-07-09T10:40:12.717Z] ] *** [2026-07-09T10:58:15.191Z] changed: [an-striker01] [2026-07-09T10:58:15.191Z] [2026-07-09T10:58:15.191Z] TASK [reboot striker01 reboot_timeout=1800] ************************************ [2026-07-09T10:58:55.553Z] changed: [an-striker01] [2026-07-09T10:58:55.553Z] [2026-07-09T10:58:55.553Z] PLAY [reconnect all nodes] ***************************************************** [2026-07-09T10:58:55.553Z] [2026-07-09T10:58:55.553Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T10:58:56.298Z] ok: [an-a01n01] [2026-07-09T10:58:56.298Z] ok: [an-a01n02] [2026-07-09T10:58:56.298Z] ok: [an-striker01] [2026-07-09T10:58:56.298Z] ok: [an-striker02] [2026-07-09T10:58:56.298Z] [2026-07-09T10:58:56.298Z] TASK [wait for node "an-striker01" to be back online _raw_params=../../common/wait_node.yml] *** [2026-07-09T10:58:56.298Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02 [2026-07-09T10:58:56.298Z] [2026-07-09T10:58:56.298Z] TASK [wait for node "all" to be back online host={{ ansible_ssh_host }}, state=started, delay=10, timeout=180] *** [2026-07-09T11:02:20.513Z] ok: [an-a01n01 -> localhost] [2026-07-09T11:02:20.513Z] ok: [an-striker02 -> localhost] [2026-07-09T11:02:20.513Z] ok: [an-striker01 -> localhost] [2026-07-09T11:02:20.513Z] ok: [an-a01n02 -> localhost] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [check nodes uptime after reboot _raw_params=if [ "{{ ansible_facts.uptime_seconds }}" -ge "{{ uptime_pre_upgrade }}" ]; then [2026-07-09T11:02:20.513Z] echo "Node current uptime higher than expected! tool failed to reboot nodes?" [2026-07-09T11:02:20.513Z] exit 1 [2026-07-09T11:02:20.513Z] fi [2026-07-09T11:02:20.513Z] ] *** [2026-07-09T11:02:20.513Z] changed: [an-a01n02] [2026-07-09T11:02:20.513Z] fatal: [an-a01n01]: FAILED! => {"changed": true, "cmd": "if [ \"629\" -ge \"628\" ]; then\n echo \"Node current uptime higher than expected! tool failed to reboot nodes?\"\n exit 1\nfi\n", "delta": "0:00:00.007356", "end": "2026-07-09 11:02:07.485789", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 11:02:07.478433", "stderr": "", "stderr_lines": [], "stdout": "Node current uptime higher than expected! tool failed to reboot nodes?", "stdout_lines": ["Node current uptime higher than expected! tool failed to reboot nodes?"]} [2026-07-09T11:02:20.513Z] ...ignoring [2026-07-09T11:02:20.513Z] changed: [an-striker01] [2026-07-09T11:02:20.513Z] changed: [an-striker02] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] PLAY [check all nodes status after upgrade] ************************************ [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:20.513Z] ok: [an-a01n02] [2026-07-09T11:02:20.513Z] ok: [an-a01n01] [2026-07-09T11:02:20.513Z] ok: [an-striker01] [2026-07-09T11:02:20.513Z] ok: [an-striker02] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [check perl-Log-Journald package version _raw_params=if ! rpm -qa | grep perl-Log-Journald | grep -q "\-99."; then [2026-07-09T11:02:20.513Z] echo "perl-Log-Journald update not found!" [2026-07-09T11:02:20.513Z] exit 1 [2026-07-09T11:02:20.513Z] fi [2026-07-09T11:02:20.513Z] ] *** [2026-07-09T11:02:20.513Z] changed: [an-a01n01] [2026-07-09T11:02:20.513Z] changed: [an-a01n02] [2026-07-09T11:02:20.513Z] changed: [an-striker01] [2026-07-09T11:02:20.513Z] changed: [an-striker02] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] PLAY [check nodes status after upgrade] **************************************** [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:20.513Z] ok: [an-a01n02] [2026-07-09T11:02:20.513Z] ok: [an-a01n01] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [check kmod-drdb package version _raw_params=if ! rpm -qa | grep akmod-drbd | grep -q "\-99."; then [2026-07-09T11:02:20.513Z] echo "akmod-drbd update not found!" [2026-07-09T11:02:20.513Z] exit 1 [2026-07-09T11:02:20.513Z] fi [2026-07-09T11:02:20.513Z] ] *** [2026-07-09T11:02:20.513Z] changed: [an-a01n01] [2026-07-09T11:02:20.513Z] changed: [an-a01n02] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] PLAY [check hosted node status after upgrade] ********************************** [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:20.513Z] ok: [an-a01n02] [2026-07-09T11:02:20.513Z] ok: [an-a01n01] [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T11:02:20.513Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T11:02:20.513Z] [2026-07-09T11:02:20.513Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T11:02:20.513Z] ] *** [2026-07-09T11:02:22.234Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T11:02:22.234Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T11:02:22.234Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T11:02:22.234Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T11:02:22.234Z] [2026-07-09T11:02:22.234Z] TASK [wait servers to be started in pacemaker _raw_params=../../common/wait_server_in_pacemaker.yml] *** [2026-07-09T11:02:22.234Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-07-09T11:02:22.234Z] [2026-07-09T11:02:22.234Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} | grep Started [2026-07-09T11:02:22.234Z] ] *** [2026-07-09T11:02:23.957Z] ok: [an-a01n02] [2026-07-09T11:02:23.957Z] ok: [an-a01n01] [2026-07-09T11:02:23.957Z] [2026-07-09T11:02:23.957Z] TASK [wait servers drbd storage to be synced _raw_params=../../common/wait_server_drbd_sync.yml] *** [2026-07-09T11:02:23.957Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-07-09T11:02:23.957Z] [2026-07-09T11:02:23.957Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced _raw_params=drbdadm status an-test-deploy{{ servernum }} | grep "disk:UpToDate" | wc -l [2026-07-09T11:02:23.957Z] ] *** [2026-07-09T11:02:24.697Z] ok: [an-a01n01] [2026-07-09T11:02:24.697Z] ok: [an-a01n02] [2026-07-09T11:02:24.697Z] [2026-07-09T11:02:24.697Z] TASK [wait servers to respond to ping _raw_params=../../common/wait_server_ping.yml] *** [2026-07-09T11:02:24.697Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-07-09T11:02:24.697Z] [2026-07-09T11:02:24.697Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping _raw_params=ping -c1 an-test-deploy{{ servernum }} [2026-07-09T11:02:24.697Z] ] *** [2026-07-09T11:02:25.437Z] ok: [an-a01n02] [2026-07-09T11:02:25.437Z] ok: [an-a01n01] [2026-07-09T11:02:25.437Z] [2026-07-09T11:02:25.437Z] TASK [wait servers to respond to ssh _raw_params=../../common/wait_server_ssh.yml] *** [2026-07-09T11:02:25.437Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-07-09T11:02:25.437Z] [2026-07-09T11:02:25.437Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh _raw_params=nc -z an-test-deploy{{ servernum }} 22 [2026-07-09T11:02:25.437Z] ] *** [2026-07-09T11:02:26.177Z] ok: [an-a01n01] [2026-07-09T11:02:26.177Z] ok: [an-a01n02] [2026-07-09T11:02:26.177Z] [2026-07-09T11:02:26.177Z] PLAY [test anvil package self update] ****************************************** [2026-07-09T11:02:26.177Z] [2026-07-09T11:02:26.177Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:28.513Z] ok: [an-a01n02] [2026-07-09T11:02:28.513Z] ok: [an-a01n01] [2026-07-09T11:02:28.513Z] ok: [an-striker02] [2026-07-09T11:02:28.513Z] ok: [an-striker01] [2026-07-09T11:02:28.513Z] [2026-07-09T11:02:28.513Z] TASK [stop anvil and scan-core services name={{ item }}, state=stopped, enabled=True] *** [2026-07-09T11:02:31.568Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-07-09T11:02:31.568Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-07-09T11:02:31.568Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-07-09T11:02:31.568Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-07-09T11:02:31.568Z] changed: [an-a01n02] => (item=scancore.service) [2026-07-09T11:02:31.568Z] changed: [an-a01n01] => (item=scancore.service) [2026-07-09T11:02:31.568Z] changed: [an-striker02] => (item=scancore.service) [2026-07-09T11:02:31.568Z] changed: [an-striker01] => (item=scancore.service) [2026-07-09T11:02:31.568Z] [2026-07-09T11:02:31.568Z] PLAY [test striker package self update] **************************************** [2026-07-09T11:02:31.568Z] [2026-07-09T11:02:31.568Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:33.290Z] ok: [an-striker01] [2026-07-09T11:02:33.290Z] ok: [an-striker02] [2026-07-09T11:02:33.290Z] [2026-07-09T11:02:33.290Z] TASK [reinstall anvil-core and anvil-striker _raw_params=yum reinstall -y anvil-core anvil-striker [2026-07-09T11:02:33.290Z] ] *** [2026-07-09T11:02:52.689Z] changed: [an-striker01] [2026-07-09T11:02:52.689Z] changed: [an-striker02] [2026-07-09T11:02:52.689Z] [2026-07-09T11:02:52.689Z] PLAY [test node package self update] ******************************************* [2026-07-09T11:02:52.689Z] [2026-07-09T11:02:52.689Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:02:53.889Z] ok: [an-a01n01] [2026-07-09T11:02:53.889Z] ok: [an-a01n02] [2026-07-09T11:02:53.889Z] [2026-07-09T11:02:53.889Z] TASK [reinstall anvil-core and anvil-node _raw_params=yum reinstall -y anvil-core anvil-node [2026-07-09T11:02:53.889Z] ] *** [2026-07-09T11:03:13.266Z] changed: [an-a01n02] [2026-07-09T11:03:13.266Z] changed: [an-a01n01] [2026-07-09T11:03:13.266Z] [2026-07-09T11:03:13.266Z] PLAY [test dr package self update] ********************************************* [2026-07-09T11:03:13.266Z] skipping: no hosts matched [2026-07-09T11:03:13.266Z] [2026-07-09T11:03:13.266Z] PLAY [test anvil package self update] ****************************************** [2026-07-09T11:03:13.266Z] [2026-07-09T11:03:13.266Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:03:13.266Z] ok: [an-striker01] [2026-07-09T11:03:13.266Z] ok: [an-striker02] [2026-07-09T11:03:13.266Z] ok: [an-a01n02] [2026-07-09T11:03:13.266Z] ok: [an-a01n01] [2026-07-09T11:03:13.266Z] [2026-07-09T11:03:13.266Z] TASK [start anvil and scan-core services name={{ item }}, state=started, enabled=True] *** [2026-07-09T11:03:15.607Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-07-09T11:03:15.607Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-07-09T11:03:15.607Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-07-09T11:03:15.607Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-07-09T11:03:15.607Z] changed: [an-a01n02] => (item=scancore.service) [2026-07-09T11:03:15.607Z] changed: [an-a01n01] => (item=scancore.service) [2026-07-09T11:03:15.607Z] changed: [an-striker02] => (item=scancore.service) [2026-07-09T11:03:15.607Z] changed: [an-striker01] => (item=scancore.service) [2026-07-09T11:03:15.607Z] [2026-07-09T11:03:15.607Z] PLAY [delete server tests] ***************************************************** [2026-07-09T11:03:15.607Z] [2026-07-09T11:03:15.607Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:03:17.328Z] ok: [an-a01n02] [2026-07-09T11:03:17.328Z] ok: [an-a01n01] [2026-07-09T11:03:17.328Z] [2026-07-09T11:03:17.328Z] TASK [check cluster is in good state _raw_params=../../common/cluster_good.yml] *** [2026-07-09T11:03:17.674Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-07-09T11:03:17.674Z] [2026-07-09T11:03:17.674Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster _raw_params=pcs status | grep Online | grep -q "{{ item }}" [2026-07-09T11:03:17.674Z] ] *** [2026-07-09T11:03:21.565Z] ok: [an-a01n01] => (item=an-a01n01) [2026-07-09T11:03:21.565Z] ok: [an-a01n02] => (item=an-a01n01) [2026-07-09T11:03:21.565Z] ok: [an-a01n01] => (item=an-a01n02) [2026-07-09T11:03:21.565Z] ok: [an-a01n02] => (item=an-a01n02) [2026-07-09T11:03:21.565Z] [2026-07-09T11:03:21.565Z] TASK [delete servers _raw_params=../../common/delete_server.yml] *************** [2026-07-09T11:03:21.909Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-07-09T11:03:21.909Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-07-09T11:03:21.909Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-07-09T11:03:21.909Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-07-09T11:03:21.909Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-07-09T11:03:21.909Z] [2026-07-09T11:03:21.909Z] TASK [check if server an-test-deploy1 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:03:21.909Z] ] *** [2026-07-09T11:03:23.632Z] skipping: [an-a01n01] [2026-07-09T11:03:23.632Z] ok: [an-a01n02] [2026-07-09T11:03:23.632Z] [2026-07-09T11:03:23.632Z] TASK [delete server an-test-deploy1 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T11:03:23.632Z] --force \ [2026-07-09T11:03:23.632Z] --server an-test-deploy{{ servernum }} [2026-07-09T11:03:23.632Z] ] *** [2026-07-09T11:03:29.691Z] skipping: [an-a01n01] [2026-07-09T11:03:29.691Z] changed: [an-a01n02] [2026-07-09T11:03:29.691Z] [2026-07-09T11:03:29.691Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:03:29.691Z] ] *** [2026-07-09T11:04:18.097Z] skipping: [an-a01n01] [2026-07-09T11:04:18.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-07-09T11:04:18.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-07-09T11:04:18.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-07-09T11:04:18.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-07-09T11:04:18.097Z] ok: [an-a01n02] [2026-07-09T11:04:18.097Z] [2026-07-09T11:04:18.097Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T11:04:18.097Z] ] *** [2026-07-09T11:04:18.097Z] skipping: [an-a01n01] [2026-07-09T11:04:18.097Z] ok: [an-a01n02] [2026-07-09T11:04:18.097Z] [2026-07-09T11:04:18.097Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T11:04:18.097Z] ] *** [2026-07-09T11:04:18.838Z] skipping: [an-a01n01] [2026-07-09T11:04:18.838Z] ok: [an-a01n02] [2026-07-09T11:04:18.838Z] [2026-07-09T11:04:18.838Z] TASK [check if server an-test-deploy2 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:18.838Z] ] *** [2026-07-09T11:04:20.559Z] skipping: [an-a01n01] [2026-07-09T11:04:20.559Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.990612", "end": "2026-07-09 11:04:20.343799", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 11:04:19.353187", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T11:04:20.559Z] ...ignoring [2026-07-09T11:04:20.559Z] [2026-07-09T11:04:20.559Z] TASK [delete server an-test-deploy2 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T11:04:20.559Z] --force \ [2026-07-09T11:04:20.559Z] --server an-test-deploy{{ servernum }} [2026-07-09T11:04:20.559Z] ] *** [2026-07-09T11:04:20.559Z] skipping: [an-a01n01] [2026-07-09T11:04:20.559Z] skipping: [an-a01n02] [2026-07-09T11:04:20.559Z] [2026-07-09T11:04:20.559Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:20.559Z] ] *** [2026-07-09T11:04:20.905Z] skipping: [an-a01n01] [2026-07-09T11:04:20.905Z] skipping: [an-a01n02] [2026-07-09T11:04:20.905Z] [2026-07-09T11:04:20.905Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T11:04:20.905Z] ] *** [2026-07-09T11:04:20.905Z] skipping: [an-a01n01] [2026-07-09T11:04:20.905Z] skipping: [an-a01n02] [2026-07-09T11:04:20.905Z] [2026-07-09T11:04:20.905Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T11:04:20.905Z] ] *** [2026-07-09T11:04:20.905Z] skipping: [an-a01n01] [2026-07-09T11:04:20.905Z] skipping: [an-a01n02] [2026-07-09T11:04:20.905Z] [2026-07-09T11:04:20.905Z] TASK [check if server an-test-deploy3 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:20.905Z] ] *** [2026-07-09T11:04:22.628Z] skipping: [an-a01n01] [2026-07-09T11:04:22.628Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.976235", "end": "2026-07-09 11:04:22.321651", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 11:04:21.345416", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T11:04:22.628Z] ...ignoring [2026-07-09T11:04:22.628Z] [2026-07-09T11:04:22.628Z] TASK [delete server an-test-deploy3 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T11:04:22.628Z] --force \ [2026-07-09T11:04:22.628Z] --server an-test-deploy{{ servernum }} [2026-07-09T11:04:22.628Z] ] *** [2026-07-09T11:04:22.628Z] skipping: [an-a01n01] [2026-07-09T11:04:22.628Z] skipping: [an-a01n02] [2026-07-09T11:04:22.628Z] [2026-07-09T11:04:22.628Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:22.628Z] ] *** [2026-07-09T11:04:22.628Z] skipping: [an-a01n01] [2026-07-09T11:04:22.628Z] skipping: [an-a01n02] [2026-07-09T11:04:22.628Z] [2026-07-09T11:04:22.628Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T11:04:22.628Z] ] *** [2026-07-09T11:04:22.628Z] skipping: [an-a01n01] [2026-07-09T11:04:22.628Z] skipping: [an-a01n02] [2026-07-09T11:04:22.628Z] [2026-07-09T11:04:22.628Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T11:04:22.628Z] ] *** [2026-07-09T11:04:22.973Z] skipping: [an-a01n01] [2026-07-09T11:04:22.973Z] skipping: [an-a01n02] [2026-07-09T11:04:22.973Z] [2026-07-09T11:04:22.973Z] TASK [check if server an-test-deploy4 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:22.973Z] ] *** [2026-07-09T11:04:24.695Z] skipping: [an-a01n01] [2026-07-09T11:04:24.695Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.982062", "end": "2026-07-09 11:04:24.198316", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 11:04:23.216254", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T11:04:24.695Z] ...ignoring [2026-07-09T11:04:24.695Z] [2026-07-09T11:04:24.695Z] TASK [delete server an-test-deploy4 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T11:04:24.695Z] --force \ [2026-07-09T11:04:24.695Z] --server an-test-deploy{{ servernum }} [2026-07-09T11:04:24.695Z] ] *** [2026-07-09T11:04:24.695Z] skipping: [an-a01n01] [2026-07-09T11:04:24.695Z] skipping: [an-a01n02] [2026-07-09T11:04:24.695Z] [2026-07-09T11:04:24.695Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:24.695Z] ] *** [2026-07-09T11:04:24.695Z] skipping: [an-a01n01] [2026-07-09T11:04:24.695Z] skipping: [an-a01n02] [2026-07-09T11:04:24.695Z] [2026-07-09T11:04:24.695Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T11:04:24.695Z] ] *** [2026-07-09T11:04:24.695Z] skipping: [an-a01n01] [2026-07-09T11:04:24.695Z] skipping: [an-a01n02] [2026-07-09T11:04:24.695Z] [2026-07-09T11:04:24.695Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T11:04:24.695Z] ] *** [2026-07-09T11:04:24.695Z] skipping: [an-a01n01] [2026-07-09T11:04:24.695Z] skipping: [an-a01n02] [2026-07-09T11:04:24.695Z] [2026-07-09T11:04:24.695Z] TASK [check if server an-test-deploy5 exists _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:24.695Z] ] *** [2026-07-09T11:04:26.415Z] skipping: [an-a01n01] [2026-07-09T11:04:26.415Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:00.934149", "end": "2026-07-09 11:04:26.079620", "msg": "non-zero return code", "rc": 1, "start": "2026-07-09 11:04:25.145471", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-07-09T11:04:26.415Z] ...ignoring [2026-07-09T11:04:26.415Z] [2026-07-09T11:04:26.415Z] TASK [delete server an-test-deploy5 from an-a01n02 _raw_params=anvil-delete-server -vv --log-secure \ [2026-07-09T11:04:26.415Z] --force \ [2026-07-09T11:04:26.415Z] --server an-test-deploy{{ servernum }} [2026-07-09T11:04:26.415Z] ] *** [2026-07-09T11:04:26.415Z] skipping: [an-a01n01] [2026-07-09T11:04:26.415Z] skipping: [an-a01n02] [2026-07-09T11:04:26.415Z] [2026-07-09T11:04:26.415Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker _raw_params=pcs status resources an-test-deploy{{ servernum }} [2026-07-09T11:04:26.415Z] ] *** [2026-07-09T11:04:26.415Z] skipping: [an-a01n01] [2026-07-09T11:04:26.415Z] skipping: [an-a01n02] [2026-07-09T11:04:26.415Z] [2026-07-09T11:04:26.415Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed _raw_params=drbdadm status an-test-deploy{{ servernum }} [2026-07-09T11:04:26.415Z] ] *** [2026-07-09T11:04:26.415Z] skipping: [an-a01n01] [2026-07-09T11:04:26.415Z] skipping: [an-a01n02] [2026-07-09T11:04:26.415Z] [2026-07-09T11:04:26.415Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed _raw_params=lvs -o name --noheadings | grep -q an-test-deploy{{ servernum }} [2026-07-09T11:04:26.415Z] ] *** [2026-07-09T11:04:26.760Z] skipping: [an-a01n01] [2026-07-09T11:04:26.760Z] skipping: [an-a01n02] [2026-07-09T11:04:26.760Z] [2026-07-09T11:04:26.760Z] PLAY [Check anvil available resources from striker01] ************************** [2026-07-09T11:04:26.760Z] [2026-07-09T11:04:26.760Z] TASK [Gathering Facts ] ******************************************************** [2026-07-09T11:04:29.095Z] ok: [an-striker01] [2026-07-09T11:04:29.095Z] [2026-07-09T11:04:29.095Z] TASK [Check anvil available resources _raw_params=echo "Waiting 5 minutes for scancore to pick up latest status" [2026-07-09T11:04:29.095Z] sleep 300 [2026-07-09T11:04:29.095Z] anvil-report-usage > /root/anvil_available_resources.stop [2026-07-09T11:04:29.095Z] diff -Naurd /root/anvil_available_resources.start /root/anvil_available_resources.stop [2026-07-09T11:04:29.095Z] ] *** [2026-07-09T11:09:38.981Z] ok: [an-striker01] [2026-07-09T11:09:38.981Z] [2026-07-09T11:09:38.981Z] PLAY RECAP ********************************************************************* [2026-07-09T11:09:38.981Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=6 [2026-07-09T11:09:38.981Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-07-09T11:09:38.981Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-07-09T11:09:38.981Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-07-09T11:09:38.981Z] [2026-07-09T11:09:38.981Z] tar: Removing leading `/' from member names [2026-07-09T11:09:38.981Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-07-09T11:09:38.981Z] /root/ci-tools [2026-07-09T11:09:38.981Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/857.tar.gz [2026-07-09T11:09:39.059Z] [Pipeline] } [2026-07-09T11:09:39.063Z] [Pipeline] // timeout [2026-07-09T11:09:39.071Z] [Pipeline] } [2026-07-09T11:09:39.074Z] [Pipeline] // script [2026-07-09T11:09:39.077Z] [Pipeline] } [2026-07-09T11:09:39.079Z] [Pipeline] // stage [2026-07-09T11:09:39.085Z] [Pipeline] stage [2026-07-09T11:09:39.086Z] [Pipeline] { (Declarative: Post Actions) [2026-07-09T11:09:39.098Z] [Pipeline] cleanWs [2026-07-09T11:09:39.148Z] [WS-CLEANUP] Deleting project workspace... [2026-07-09T11:09:39.148Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-07-09T11:09:39.172Z] [WS-CLEANUP] done [2026-07-09T11:09:39.179Z] [Pipeline] script [2026-07-09T11:09:39.184Z] [Pipeline] { [2026-07-09T11:09:39.217Z] [Pipeline] stage [2026-07-09T11:09:39.218Z] [Pipeline] { (Copying logs) [2026-07-09T11:09:39.274Z] [Pipeline] node [2026-07-09T11:09:39.413Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-07-09T11:09:39.414Z] [Pipeline] { [2026-07-09T11:09:39.476Z] [Pipeline] node [2026-07-09T11:09:39.645Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-07-09T11:09:39.645Z] [Pipeline] { [2026-07-09T11:09:39.650Z] [Pipeline] sh [2026-07-09T11:09:39.907Z] + mkdir -p /var/lib/jenkins/locks [2026-07-09T11:09:39.922Z] [Pipeline] echo [2026-07-09T11:09:39.922Z] RWLock: FD for lock log_archive in stage projectFinishUp is 521 [2026-07-09T11:09:39.926Z] [Pipeline] sleep [2026-07-09T11:09:39.926Z] No need to sleep any longer [2026-07-09T11:09:39.929Z] [Pipeline] echo [2026-07-09T11:09:39.929Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-07-09T11:09:39.933Z] [Pipeline] } [2026-07-09T11:09:39.935Z] [Pipeline] // node [2026-07-09T11:09:39.939Z] [Pipeline] echo [2026-07-09T11:09:39.939Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/857/ [2026-07-09T11:09:39.940Z] [Pipeline] sh [2026-07-09T11:09:40.196Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/857/ [2026-07-09T11:09:40.199Z] [Pipeline] sh